6 Schema Architecture

The page-language core and how collection grammars extend it

6.1 The canonical page DTD (v4.0) and its core

The canonical page grammar lives at pdai/pipeline/ai-scholar/src/ai_scholar/schemas/page.dtd. Since v4.0 it includes two modules shared by every PDAI grammar (Chapter 2):

Module	Declares
`pdai-core.mod`	`<marginal>` with the closed type list (markers `line-number`, `folio`, `symbol`; notes `topic`, `reference`, `date`), `<meta>` with the partitur stamp, `<figure>` / `<caption>` / `<description>`, the renditions of `<hi>`
`pdai-table.mod`	`<table>`, `<tr>`, `<th>`, `<td>` (never flattened to `<p>`)
`page.dtd`	`<page>`, `<header>`, `<main>`, `<p>`, `<fn>`, `<parno>`, `<hi>`, `<br>`, `<footnotes>`, `<footnote>`, `<footer>`

Every content model of page.dtd is a parameter entity (page.content, main.content, p.content, hi.content, fn.model, header.model, footnotes.model, footnote.content …). The page variants (page_letter.dtd v4.2, page_section.dtd, page_annotated.dtd, page_translated.dtd) are wrappers of page.dtd; the record grammars (letter.dtd, section.dtd, parno.dtd, the collections’ tablet.dtd, catalogue.dtd and KGW 19’s document.dtd) include pdai-core.mod.

6.2 Layer 1: Generic ai-scholar stage chain

The ai-scholar processing pipeline produces pages against page.dtd. Its OCR templates carry the page-language fragment as a generated block (python -m ocr_runner.page_language sync-ai-scholar). This is the layer that enestroem, Euler and the other unprofiled collections use.

6.3 Layer 2: Collection wrappers

A collection with elements that have no canonical equivalent has a wrapper config/page.dtd: it sets the entities it needs, declares its own elements and attributes, and includes page.dtd. The first declaration of an entity or attribute wins, so the overrides stand before the include.

Important

A collection grammar adds to the page language. It never removes or renames a core element or margin type, and it does not declare record elements (cross-page assembly belongs to the record grammar). It may add margin types (marginal-type.extra), widen what a margin or a cell may contain (marginal.content, cell.content) and relax meta / figure requirements (meta.req, figure.ref.req).

Collection	Its own elements (beyond the page language)
`kepler-prim`	`running-header`, `letter-head` / `letter-closing` families, `section`, `lineno`, `fnmark`, `equation`; KGW 19: `document-entry`, `doc-*`, margin type `doc-id`
`copernicus-prim`	`apparatus` / `app`, `toc` / `toc-entry` / `label` / `title` / `pageref`; `fn` is empty with `@n`
`diaries`	`block` with `field`, `witness`, `copy`, `lbat`, `photo`, `bibliography`, `date`, `coverage`, `remark`, `comment`, `ref`, `row`; typed `section`; legacy `number`, `apparatus`, `comments`, `calendar`
`medieval-sec`	`entryblock` with the catalogue fields, `register` / `row` / `term` / `gloss`, `supplied`, `ref`; margin type `entry-number`
`pirckheimer`	legacy `blocks` / `block` grammar (production pages use `page_letter.dtd`)
letter collections	`page_letter.dtd` v4.2 wrappers (pirckheimer, kepler-prim KGW 13–16, kepleriana-munich-2024, eulerOO-IVA, karman-nachlass)

The flattened copy that deployment writes to PDAI-1 is what the viewer host and external readers see; the audit validates against the sources.

6.4 The page / record split

A page DTD governs one physical page. Records assembled across two or more pages belong to a separate record DTD. This split keeps truncation visible: a page that begins mid-letter and ends mid-paragraph is valid XML against the page grammar; the assembler joins it (ocr_runner.assemble_letters for letters).

page_letter.dtd → one physical scan page (letter-part per letter on it)
letter.dtd      → one assembled letter from first to last page, <pb/> between pages

Never declare the record root element as a required top-level child of <page>. Use optional children so a page with an open record at top and bottom is still DTD-valid.

6.5 Versions

page.dtd v4.0, page_letter.dtd v4.2, letter.dtd v4.2, pdai-core.mod v1.0, pdai-table.mod v1.0 (schemas/registry.yml). Collection wrappers name the version of the grammar they wrap in their header comment; page XML names it in <meta output_schema="…">.

6.6 Agent-driven schema derivation

New collection schemas are derived by schema_derivation_agent (Partitur at development/ai-agent-orchestrator/agent-store/schema_derivation_agent/, registered at pipeline/ocr/schema-derive).

The agent has four stages:

locate-conventions — reads the edition’s front-matter statement of record form; stops if no declaration exists.
emit-record-schema — writes config/<record>_schema.yaml with status: front-matter | facsimile-only grades for every field.
verify-against-facsimile — checks the draft against 3–4 body pages and records corrections.
emit-dtds — generates the collection wrapper config/page.dtd (entity overrides + own elements + include of page.dtd) and config/<record>.dtd.

All four stage instructions are externalized as SKILL.md files under pdai/skills/pipeline/schema-derive/.