4  One grammar per document

A collection is a set of documents, and its documents differ in format. The grammar, the prompt layer and the model are therefore fixed per document, in a fixed order, and each step is measured before the next begins.

Until version 1.1 of the Partitur pilot-gated-ocr-pipeline, a document took the grammar of its collection. That holds where the volumes of a collection share one format (the volumes of one critical edition). It fails where they do not: within the collection Kirch, the Forschungsberichte are modern prose, the Handbuch is a sequence of articles with fixed rubrics, and the Kalenderreihen are calendars whose pages are tables in two colours. Version 1.2 makes the document the unit.

TipThe order

Profile the document, derive its grammar, train the layer for one engine, check the fit against the facsimile, select the model, run, certify. A later step never begins on the output of an earlier step that has not passed.

G scale assess-document-scale > 50 pages: gate mandatory profile profile-document range of page.dtd, page classes, entry types scale->profile grammar derive-document-grammar config/volumes/<doc>/page.dtd, <class>.dtd profile->grammar pilot draw-pilot-sample, execute-pilot-ocr, audit-structural-gate grammar->pilot train train-layer <prefix>.<doc|class>.<engine>.j2 pilot->train fit fit-check reading against facsimile train->fit fit->train fail select select-model best measured engine with a passed layer fit->select run unlock-and-execute-full budgeted batches, one runner select->run certify certify-and-learn pdai-audit --all --emit-annotations run->certify
Figure 4.1: The process of Partitur pilot-gated-ocr-pipeline v1.2. Solid edges are the order of execution; the dashed edge returns to layer training when the fit check reports a failure.

4.1 What the profile establishes

The profile of a document (config/volumes/<doc>/document_profile.yaml, skill document-profile) records three things, each with an evidence grade.

Question Answer Where it is used
Which part of the universal DTD does the document use, and what does it need beyond it? element families in use; extension margin types (marginal-type.extra) and renditions (hi-rend.extra) grammar derivation
Which page classes does the document contain? class name, page count, specimen pages page-class grammar, class layers, fit specimens
Which entry types make up its structure? name, XML element, printed opener, independent count where stated segmentation pass

The grades are those of the schema-first method: front-matter (stated in the edition), facsimile-only (observed, provisional), unattested (neither). A vocabulary item graded unattested is not declared in the grammar.

Page OCR remains page-scoped. Entry types are the input of a separate segmentation pass that assembles records across pages; they do not change what one page reading contains.

4.2 The grammar of one document

A document grammar is a wrapper of page.dtd v4.0 (Chapter 2): it overrides content-model parameter entities, then includes the universal DTD. The first declaration of a parameter entity wins, so the override comes first. Extension vocabulary enters only through the two extension entities.

Extension Entity Example
margin type beyond the closed list marginal-type.extra doc-id in KGW 19
hi rendition beyond the core list hi-rend.extra \| red for the red type of a calendar

ocr_runner.page_language.page_dtd(collection, doc, page_class) resolves the grammar in this order and returns the first that exists:

  1. config/volumes/<doc>/<page_class>.dtd, when a page class is given;
  2. the DTD named by config/audit_rules_<doc>.yaml;
  3. the DTD named by the collection’s config/audit_rules.yaml;
  4. the universal page.dtd.

The page class is stamped on the page (<meta page_class>), so the class grammar and the class layer can be recovered from a saved page.

4.3 The layer for one document and one engine

ocr_runner.cure_config.compose builds the prompt from layers and stops at the last layer that exists. With a page class, four layers are added after the volume-and-engine layer.

Order Layer file
1 page-language fragment (filled from the document’s grammar)
2 <prefix>.base.j2
3 <prefix>.<volume>.j2
4 <prefix>.<engine>.j2
5 <prefix>.<volume>.<engine>.j2
6 <prefix>.<class>.j2, <prefix>.<class>.<engine>.j2
7 <prefix>.<volume>.<class>.j2, <prefix>.<volume>.<class>.<engine>.j2

The stamp <meta partitur="<prefix>@<sha12>" layers="…" overlay="…" page_class="…"> names every layer that was applied. A layer trained for one engine does not qualify another: each candidate engine is trained separately, with the A/B baseline measured on the unpatched layer first (skill ocr-instruction-learning).

4.4 The fit check

python -m ocr_runner.fit_check compares a reading with its facsimile. Every finding has a severity; a page fits when it has no fail.

Code Compares fail when
FIT-DTD reading against the document’s grammar the reading is invalid
FIT-AUDIT collection audit rules the audit reports a blocker
FIT-TABLE-FLAT rows of cells in the image against table in the reading the image has at least four rows of cells and the reading has no table
FIT-TABLE-ROWS row count, image against reading the counts differ by more than 45 % (warn above 25 %)
FIT-TABLE-COLS modal column count the modal columns differ by three or more (warn from one)
FIT-MARGIN margin items of the image against marginal an item of the image is missing
FIT-TEXT word agreement with the text witness agreement is below 70 % (warn below 85 %)
FIT-FIGURE figure zones against figure never; warn only

The image side is the geometry of Apple Vision (fastimage --mode ocr): the lines of a page with their position. It is a witness for structure everywhere, and for text only where it reads the script. The profile therefore records the text witness: vision for antiqua, a second reading for Fraktur and special scripts (dual_read.divergences).

The check also writes a review sheet per page: the facsimile beside the reading, rendered with pdai-page.css. The sheet is the input of the visual audit (skill visual-layout-audit), which attributes a remaining flaw to one layer — prompt layer, normalizer, CSS or DTD — before anything is changed. The sheet uses a structural rendering, not the viewer; the acceptance view remains the viewer on the document’s port.

4.5 Model selection

The engine of the full run is the best measured engine among those with a passed layer: quality first, then latency, then cost. The measurements come from the Model Assessment Hub (ma_best_model), and each pilot run is recorded with ma_record_run. The default is kimi-k2.7-code:cloud; gemini-3.8-flash with thinking off runs only for a page on which Kimi failed. A different default is reported to the owner and is not adopted without confirmation.

4.6 Registered contract

The stages profile-document, derive-document-grammar, fit-check and select-model are registered in partitur_register.duckdb under the goal path pipeline/ocr/pilot-gated, with the result specification and verification of each stage (production pipeline). The former stage improvement-loop is now train-layer.