PDAI Structured-XML OCR System

How a PDAI collection is folded on disk — the plan, the metadata, and the two DTDs

Author

Gerd Graßhoff

Published

September 29, 2026

A PDAI collection is not a pile of PDFs. It is a folder of documents plus a small set of files that say what belongs here, how each document is identified, and what grammar its pages must obey. Those files are the metadata. Without them the pages can still be read; they cannot be cited, scoped, or checked.

TipIn one sentence

The collection plan names the documents; each document’s biblio.xml is the bibliographic source of truth; the collection .bib is rebuilt from those files and is never hand-edited; the DTDs on the collection say what a page and a record look like.

0.1 Two trees, one collection

Code and data are separate on purpose. The workspace at ~/ai-mac-studio/pdai/collections/<name>/ holds schemas, scripts and instructions. The documents themselves live on the PDAI-1 volume at /Volumes/PDAI-1/pdai-collections/<name>/. Nothing in the code tree holds page images; nothing on the volume is the place to invent a schema.

The volume folder is what this chapter describes. It has two layers.

/Volumes/PDAI-1/pdai-collections/<name>/
├── volumes.yaml              the plan: which documents belong here
├── bibliography.yaml         verified title-page records (optional source)
├── <name>.bib                rebuilt from per-document biblio.xml
├── page.dtd                  grammar of one OCR'd page
├── <record>.dtd              grammar of the assembled record
├── status.json               last contract audit
└── <biblabel>/               one folder per document
    ├── <biblabel>.pdf
    ├── biblio.xml            bibliographic metadata (source of the .bib)
    ├── pdai-config.json      harvest + processing record
    └── pages/page_NNN/

The folder name of a document is its citation key. Bischoff2004/ produces @book{Bischoff2004, …}. That is the join the search stack uses; renaming the folder without rebuilding the bib breaks it.

0.2 A real collection

medieval-sec is Bischoff’s Katalog der festländischen Handschriften des neunten Jahrhunderts — three volumes, nothing else. This is the folder as it stands on PDAI-1.

The collection after the metadata was set.

medieval-sec/
├── volumes.yaml
├── bibliography.yaml
├── medieval-sec.bib          3 entries, rebuilt by bib_build
├── page.dtd
├── catalogue.dtd
├── entry_schema.yaml
├── status.json               overall: not_ready
├── Bischoff1998/             Teil I: AACHEN – LAMBACH, 1998
├── Bischoff2004/             Teil II: LAON – PADERBORN, 2004
└── Bischoff2014/             Teil III: PADUA – ZWICKAU, 2014

Plan volumes.yaml lists exactly those three folders. Curtze is not in the plan, so it is not in the collection — even if an older bib once named it.

diaries uses the same layout with different names (tablet.dtd instead of catalogue.dtd, ten ADART volumes instead of three Bischoff volumes). The files change; the jobs do not.

0.3 The plan

volumes.yaml is the collection plan. It answers one question: which documents belong here, and how is each one processed? It does not describe the printed record — that is the schema — and it does not carry bibliographic fields. A document that is not in the plan is not in the collection.

The three Bischoff volumes, as the plan states them.

collection: medieval-sec
collection_root: /Volumes/PDAI-1/pdai-collections/medieval-sec

volumes:
  - id: Bischoff1998
    volume: I
    teil: "AACHEN – LAMBACH"
    route: scan
    content: catalogue
    pages: 518
    source: /Volumes/PDAI-1/pdf_sortiert/bischof/Band_I_Einzelseiten.pdf

id is the folder name. route and content tell the pipeline what kind of pages to expect. source is the PDF the folder was harvested from — a provenance pointer, not a second copy of the book.

diaries adds split: true for volumes photographed as two-page spreads. That is a plan fact: it changes how pages are rendered, not what a tablet is.

0.4 Bibliographic metadata

A document’s identity is written once, in biblio.xml, and then derived everywhere else. The file conforms to biblio.dtd in pdai/pipeline/ai-scholar/src/ai_scholar/schemas/. Required fields are entrytype, biblabel, at least one author, title, and year. Everything else is optional and is omitted when the printed source does not give it.

WarningNever invent a bibliographic field

Every value in biblio.xml is read off the title page or the imprint / CIP page of that volume. A field the book does not print stays absent. Band I of Bischoff does not name an editor on the title page, so Bischoff1998 has no <editor> — even though Bands II and III name Birgit Ebersperger.

Band II, taken from the title page (PDF p.001) and the imprint (PDF p.002).

<biblio>
  <entrytype>book</entrytype>
  <biblabel>Bischoff2004</biblabel>
  <author>Bischoff, Bernhard</author>
  <editor>Ebersperger, Birgit</editor>
  <title>Katalog der festländischen Handschriften des
         neunten Jahrhunderts (mit Ausnahme der wisigotischen)</title>
  <subtitle>Teil II: LAON – PADERBORN</subtitle>
  <series>Veröffentlichungen der Kommission für die Herausgabe
          der mittelalterlichen Bibliothekskataloge
          Deutschlands und der Schweiz</series>
  <year>2004</year>
  <publisher>Harrassowitz Verlag</publisher>
  <address>Wiesbaden</address>
  <isbn>3-447-04750-X</isbn>
  <language>de</language>
</biblio>

biblabel matches the folder name. The half-title of this volume prints the Bayerische Akademie der Wissenschaften as the issuing body of the series, not as the author — so the academy is not in <author>.

bibliography.yaml in the collection (and, for the code tree, in pdai/collections/<name>/config/) is the working record of those verified fields, with a _provenance note naming the printed page. It is the file a setup script reads when it writes biblio.xml. It is not a second bibliography and it is not cited.

The collection file <name>.bib is generated:

python -m ai_lemma_index.stanzadoc.bib_build --collection medieval-sec

bib_build walks every document folder, reads biblio.xml, and writes one BibLaTeX entry whose key is the folder name. Documents without biblio.xml get no entry — a gap, not a guessed record. Hand-editing the .bib is overwritten the next time the command runs.

pdai-config.json is a different kind of metadata. It records how the document was harvested and processed: source PDF path, page count, OCR model, collection name, and — once the schema exists — a schema_ref pointing at the DTDs on the collection. It is not the bibliography. Correcting a title means editing biblio.xml and rebuilding the bib, then updating the document block in pdai-config.json so the two do not drift.

0.5 The two DTDs

A formalised edition prints structured records inside running text. Those records span page breaks, so the collection carries two grammars, not one.

File Governs May do Must not do
page.dtd one OCR’d page record what is printed here, as complete or cut blocks infer a number, a heading, or a field from another page
<record>.dtd the assembled record join a block cut at the bottom of one page to the block cut at the top of the next invent a field the pages never printed

The record is a catalogue entry. catalogue.dtd groups entries under a <libraryplace> and lets <shelfmark> repeat, because a manuscript dispersed across libraries is printed as a chain joined by +.

page.dtd          →  <page> of <entryblock> elements
catalogue.dtd     →  <catalogue> of <libraryplace> / <entry>

The record is a tablet edition. tablet.dtd on the collection (and in config/) covers both the page and the assembled <tablets> document.

page.dtd          →  <page> of diary / comments / calendar blocks
tablet.dtd        →  <tablets> of <tablet> records

Both DTDs sit on the collection as well as in the code tree, so a page XML file can name them without reaching back into ~/ai-mac-studio. pdai-config.json records the paths under schema_ref.

The schema YAML (entry_schema.yaml / tablet_schema.yaml) is the prose behind the DTD: every element cites the printed page it came from. How that schema is derived, and why it is written before production OCR, is the schema-first method.

0.6 The stylesheet pair

A page.dtd can only be rendered meaningfully if the viewer knows how each element should look. The rendering convention is therefore a pair:

collections/<name>/config/   page.dtd  +  page.css    authored here
/Volumes/PDAI-1/pdai-collections/<name>/  page.dtd + page.css    deployed here

Both files of the pair are wrappers of the shared page language (Chapter 2): the collection page.dtd includes page.dtd v4.0 and pdai-core.mod, and the collection page.css is an overlay on the shared book sheet. The deployed page.dtd is a flattened, generated copy.

0.6.1 How it works

Schema rendering is always on. pdai_server.py injects the shared sheets (pdai-page.css, pdai-epistolary.css, pdai-astronomical.css, pdai-translation.css, pdai-receipt.css), the renderer repro-xml.js and window.PDAI_SCHEMA_RENDER = true into every viewer page, and adds the collection’s page.css (served at GET /api/collection/{collection}/page.css) as the last sheet when the collection has one. A collection without page.css is rendered by the shared sheets alone.

The renderer (ReproXML) parses the page XML with DOMParser(…, 'text/xml'), so empty DTD elements (<witness/>, <row/>, <date/>) survive as empty nodes. Every element becomes an HTML element with data-el="tagname" and each XML attribute as data-<attribute> (data-type, data-side …); a marginal also carries data-kind (marker or note), which the margin rules of pdai-page.css use.

0.6.2 Visual language

Property Value
Ink #1a1a1a on white
Accent / hairlines Oxford blue #002147 only
Typeface Georgia / Times New Roman (always present, no load promise)
Uppercase None in content; labels use lower-case content: "…"
Structure Indentation, vertical rhythm, hanging numbers, 1 px hairlines
Quiet labels 10 px #8a8a8a sans labels via ::before { content: attr(…) }

The base rules covering elements common to every DTD (page, header, p, hi, table, figure, footnotes, …) live in ai-viewer/config/styles/pdai-page.css. Collection-specific rules override and extend that base.

0.6.3 Coverage tool

ai-viewer/scripts/css_coverage.py parses the DTD and the collection CSS and reports any element or enum value with no rule. It exits non-zero on a gap and accepts --scaffold to emit a starter stylesheet:

# Verify coverage
python scripts/css_coverage.py ../collections/diaries/config

# Generate a skeleton for a new collection
python scripts/css_coverage.py ../collections/new-collection/config --scaffold > page.css

Run the coverage check whenever the DTD gains a new element or attribute.

0.6.4 Iron rule: DTD and stylesheet authored together

A new page DTD must have a matching page.css before the collection enters production OCR. The coverage check enforces this at CI time. Unstyled elements render as anonymous <div> blocks, which is not an error — but they will never look correct. Author the two files in the same session and commit them together.

0.7 What is not metadata

Pages, OCR XML, structure.xml, the lemma index, the stanza database and the Chroma store are products. They are absent or incomplete on a collection that is not_ready, and that is a processing fact, not a metadata gap. status.json reports those products. It does not replace the plan.

status.json is written by the collection contract audit. Correcting a document count after a volume is removed is honest; claiming the collection is ready because the bib now validates is not.

0.8 The -prim/-sec collection convention

When a collection contains both primary sources and secondary literature, split them into two separate collection folders on PDAI-1:

  • <name> — primary sources (correspondence originals, critical editions of the author’s own works, archival documents)
  • <name>-sec — secondary literature (Jahrbücher, monographs, Festschriften, research volumes about the person/topic)

Both are served by the same edition viewer and appear under the same edition in editions.yaml. The split uses ditto — never symlinks.

# Example: Pirckheimer
pirckheimer/         # Willibald Pirckheimers Briefwechsel (7 vols., Beck)
pirckheimer-sec/     # Pirckheimer-Jahrbuch, edited volumes, secondary scholarship

Why split? Different material classes → different OCR templates → different DTDs. A printed scholarly journal must never be transcribed with a letter template. The -sec split makes the classification boundary explicit in the data structure itself, so ai-classifypdf prepare <collection> can apply the right profile per collection without any guesswork.

Existing examples: kepler-prim/kepler-sec, copernicus-prim/copernicus-sec, waefler-prim/waefler-sec, euler_sec, pirckheimer/pirckheimer-sec.

config/ artifact set (per collection, in pdai/collections/<name>/config/):

artifact purpose
collection_profile.yaml OCR run plan: class, template, DTDs, run_approved
ocr_classification.json per-document classification result + evidence
bibliography.yaml verified title-page records with _provenance
volumes.yaml collection manifest: which documents belong, what type

These are produced by ai-classifypdf prepare <collection> (the preparation step) and consumed by OCR automatically. They are the PDAI equivalent of “metadata before data”.

0.9 Iron rules

WarningSix constraints that keep the layout honest
  1. The plan is the membership list. A bib entry without a folder, or a folder without a plan row, is a defect. Do not leave either standing.
  2. biblio.xml is sourced, never guessed. Title page and imprint only. bib_build will not invent the missing year.
  3. Folder name = bib key = biblabel. Three spellings of the same identity, kept identical.
  4. The page DTD does not assemble records. Cross-page join is the record DTD’s job, and only after the pages exist.
  5. Classification precedes OCR. Run ai-classifypdf prepare <collection> before any production OCR. Never hand-pin a template in pdai-config.json without an ocr_classification.json that documents why.
  6. Pilot gate for book-sized documents (> 50 pages). For large documents, never launch full batch processing directly. Draw a 5-page random sample, execute pilot OCR, and audit against DTD/audit rules/facsimile. If defects emerge, run the improvement loop (prompt overlay, normalizer regex, document-specific page.css) until 100% PASS before batch unlocking.

The production commands that fill pages/ are in the pipeline skill. This chapter stops at the folder that those commands write into.