21 Universal letter DTD
One page grammar for the letter editions pirckheimer, Kepler KGW 13–16, eulerOO-IVA and karman-nachlass
21.1 Purpose
Four collections contain letters: the Pirckheimer Briefwechsel (pirckheimer), the letter volumes 13–16 of the Kepler Gesammelte Werke (kepler-prim), the Euler correspondence volumes of series IVA (eulerOO-IVA) and the Kármán archive (karman-nachlass). Each was transcribed under its own page grammar. A query about the sender, the date or the closing of a letter therefore needed one reader per collection. The universal letter DTD page_letter.dtd (v4.2 since 2026-09-28; pdai/pipeline/ai-scholar/src/ai_scholar/schemas/) is one page grammar that holds every letter convention attested in these editions. It is a wrapper of the canonical page.dtd v4.0: margins, meta, figures, tables, p, hi, fn and footnotes come from the page language (Chapter 2), and this file adds the letter families. v4.1 added the diplomatic marks del / add.
21.2 Derivation
The grammar was derived per collection and then unified; no element was declared from general expectations.
Per collection — the schema-derive Partitur read the printed front matter of each edition and a sample of body pages and wrote a
config/letter_schema.yaml. Each section carries an evidence grade:front-matter(the edition’s own editorial principles state the convention),facsimile-only(observed on pages, not stated) orunattested.Collection Sections front-matter facsimile-only pirckheimer 20 0 20 eulerOO-IVA 17 14 3 karman-nachlass 16 0 16 Union — the unify-letter-schema instruction merged the three schemas into
letter_union.yaml: 34 elements, 0 rejected, 15 unattested generic page elements, 9 excluded.Grammar —
page_letter.dtddeclares the union. Each letter element carries a provenance comment naming the collections that attest it and the evidence grade.tests/test_letter_dtd_union.py(ocr-runner) checks both directions: every union element is declared, and every declared letter element is in the union or in the listed structural containers.
21.3 The page structure
A page holds its content in main. A letter’s share of the page is one letter-part; a page on which two letters meet holds two.
| Element | Content |
|---|---|
letter-part[nr, partial] |
the part of one letter on this page, in page order; partial="start" when the letter continues on the next page |
letter-head |
reference, sender, addressee, place, date, dateline, archive, witness, regest, printed letterhead, file reference, attention line |
salutation, p |
salutation and letter text |
letter-closing |
valediction, place-date, signature, postscriptum, address, endorsement, enclosure |
enclosure, translation[lang] |
enclosed documents and printed translations |
footnotes / footnote |
the edition’s apparatus on this page |
register[type] |
non-letter matter printed in a letter edition: registers, indexes, afterword commentary |
letter-starts / letter-ends with letter-ref[nr] |
page-level list of the letters that begin or end on the page |
main admits no letter-head outside a letter-part or register: a head without its letter is a structural defect.
The record grammar letter.dtd (v4.2) describes one assembled letter across pages; it replaces anhang by enclosure | translation. Its %inline; carries the diplomatic marks del, add, supplied since v4.2 (before, the record’s entity silently overrode the page grammar’s and Kepleriana records with these marks were invalid).
21.4 One assembler
Records are assembled by python -m ocr_runner.assemble_letters --collection <c> --doc <d> --write for every collection whose pages are v4: consecutive letter parts with the same nr are one letter, nr="cont" continues the open letter, nr="anhang" is its enclosure, and a page boundary inside a letter becomes <pb n="printed page" pidx="…"/>. Records and letters_assembled.jsonl go to <data>/<doc>/segments/letters/. Run of 2026-09-28: KGW 13–16 631 letters (442 multi-page), pirckheimer 233 (182), kepleriana-munich-2024 59 — all valid. Pirckheimer’s legacy assembler (letter.dtd v1) is retired.
21.5 Collection overlays
Each collection receives two overlays in its config/ folder, page_letter.dtd and letter_record.dtd. Both include the universal file as an external entity; a collection-specific restriction is declared before the inclusion, since the first declaration of an entity binds. Flat pages (main > p) of eulerOO-IVA and karman-nachlass are valid v4 pages; their letter markup is added only through the gated pilot (see Flat collections). karman-nachlass has an audit_rules.yaml naming its page_letter.dtd since 2026-09-28.
21.6 Transformation of the structured collections
pirckheimer and KGW 13–16 already carried letter structure in another form. A declarative mapping (config/letter_mapping.yaml) and the normalizer letter-universal (ocr_runner.normalize_letter) convert it. The normalizer never changes the order of the page text; its ID contains the hash of the mapping file, so a changed mapping invalidates every earlier gate.
Each document passed the specimen gate of ocr_runner.normalize_apply on five pages before any page was written. Every written page is backed up as .vN.
| Document | Letter pages | Register pages | Broken anchors |
|---|---|---|---|
| pirckheimer korrespondenz | 526 | — | 0 |
| KGW 13 | 369 | 59 | 0 |
| KGW 14 | 421 | 63 | 0 |
| KGW 15 | 479 | 69 | 0 |
| KGW 16 | 390 | 75 | 0 |
39 KGW pages were not converted, because they are invalid against Kepler’s own earlier page DTD already (undeclared MathML math/mrow, misplaced elements). They remain in the register as SYN-DTD for the cure.
21.6.1 Afterword pages
The first audit after the transformation found 15 unchanged afterword pages (running head NACHBERICHT) invalid against v4.0; 13 of them had been valid under the old grammar. The OCR had set the commentary heading for each letter as letter-head directly in main. An afterword page is non-letter editorial matter; the mapping now names such pages in register_page_if, and the normalizer places their content unchanged in register[type] (nachbericht, register, verzeichnis). The page reader of the lemma index returns identical text for all 266 register pages before and after.
21.7 Audit after the transformation
The collection audit rules name the v4.0 overlays (audit_rules.yaml for pirckheimer, audit_rules_kgw-13.yaml … audit_rules_kgw-16.yaml for the KGW letter volumes; pdai-audit --vol selects the volume file when it exists).
| Volume | Pages | Pages with repair defects | SYN-DTD pages |
|---|---|---|---|
| pirckheimer korrespondenz | 526 | 3 | 0 |
| KGW 13 | 446 | 11 | 0 |
| KGW 14 | 516 | 57 | 24 |
| KGW 15 | 567 | 29 | 9 |
| KGW 16 | 478 | 50 | 6 |
The SYN-DTD pages are exactly the 39 excluded pages. For KGW 15 an audit register from before the transformation exists: pages with repair defects fell from 52 to 29, SYN-DTD pages from 32 to 9, and no page received a defect code it did not have before. For the other volumes no earlier register was kept.
The vocabulary rules of the new grammar (SCH-WITNESS-TYPE, SCH-FNTYPE-UNKNOWN, SCH-REGISTER-TYPE, SCH-FOREIGNLANG-UNKNOWN) fire on no page. The structural codes without a layer received layer proposals (quality/attribute_proposals.json); none is in force until the owner confirms it.
21.8 Flat collections
eulerOO-IVA and karman-nachlass carry letter text in plain paragraphs without letter structure. A mapping cannot convert them; the letter structure has to be produced by a model. The Partitur letter-flat-pilot compares two routes on five body pages per document:
- (a) overlay re-OCR — the page image is transcribed again under the letter-ocr-overlay instruction;
- (b) restructuring — the existing page XML is restructured into v4.0 by a text model; the word sequence must remain unchanged.
No volume is processed before the owner’s decision.
21.8.1 Pilot result (26 September 2026)
Both routes ran on kimi-k2.7-code:cloud via Ollama with four concurrent calls and without the Gemini fallback (PDAI_CURE_FALLBACK=off); every answer came from Kimi. Run trees: /Volumes/PDAI-1/tmp/schema-derive/<collection>/pilot1/.
| Collection | Route | DTD-valid | Word sequence kept | Mean word agreement | Pilot pass |
|---|---|---|---|---|---|
| eulerOO-IVA | (a) re-OCR | 0/5 | — | 0.994 | 0/5 |
| eulerOO-IVA | (b) restructuring | 5/5 | 5/5 | 1.000 | 2/5 |
| karman-nachlass | (a) re-OCR | 2/5 | — | 0.977 | 2/5 |
| karman-nachlass | (b) restructuring | 0/5 | 5/5 | 1.000 | 0/5 |
Three of the eulerOO-IVA sample pages are no letter pages (introduction p. 825, subject index p. 1144, section title “VII ANHÄNGE”). Route (b) set no letter part on them and placed the index in register; the pilot’s pass condition (at least one letter part) counts these correct answers as failures. Measured by content, route (b) is correct on 5 of 5 eulerOO-IVA pages. Route (a) failed on eulerOO-IVA because the model introduced header elements the grammar does not declare (running-header, parno, p in header).
On karman-nachlass neither route passed in run 1. The errors of route (b) were content-model violations — attributes the grammar does not declare (marginal@place, file-reference@place) and children of letter-part in an order the grammar does not admit; the words are kept on every page.
21.8.2 Pilot 2: karman-nachlass through the page-language partitur (28 September 2026)
Same five pages, kimi-k2.7-code:cloud, no Gemini fallback. Route (a) uses the composed partitur karman-letter-ocr (page language + karman-letter-ocr.base.j2 + karman-letter-ocr.kimi.j2): the base maps the collection’s convention names (line, body, closing, marginal_note …) onto the universal elements and lists the admitted children of every letter element, read from the DTD; the Kimi overlay names the run-1 errors. Route (b) gives the restructuring instruction the same grammar rows, convention map and page-language fragment. The word measure now counts a <br/> as a word boundary (before, Professor<br/>Th. became one word and failed the word-sequence gate); <lb/> is normalized to <br/>. Run tree /Volumes/PDAI-1/tmp/schema-derive/karman-nachlass/pilot2/, summary collections/karman-nachlass/quality/letter-pilot-2026-09-28/pilot.json.
| Route | DTD-valid | Letter parts | Margin-type defects | Word sequence kept / agreement | Pilot pass |
|---|---|---|---|---|---|
| (a) re-OCR, composed partitur | 5/5 | 5/5 | 0 | agreement 0.977 (≥ 0.95 on 5/5) | 5/5 |
| (b) restructuring | 5/5 | 5/5 | 0 | 2/5 exact, agreement 0.987 | 2/5 |
Route (a) passes the gate on all five pages; its word differences to the existing transcription are mostly joined line-end hyphenations and diacritics the old reading lacked. No mapping is registered and no volume is converted: route (a) is a re-OCR, not a deterministic remap, and the owner decides whether karman-nachlass is re-read under this partitur.
The sample draw found one eulerOO-IVA page with a facsimile but no page XML (978-3-319-33989-4/page_752); it was replaced by the next drawn page.
21.9 Viewer
The viewer renders the letter elements through the book-sheet renderer window.ReproXML (config/scripts/repro-xml.js; xmlToStructuredHtml in viewer.html is only its fallback), which sets data-el and data-<attr> on each element; the collection’s page.css styles them by [data-el="…"] selectors only and takes its colours from the variables of pdai-page.css, so that the light and the dark scheme both stay legible. letter-ref is not displayed; it is navigation data. The letter-head handler renders the head’s reference chips and then every further child in document order.