flowchart LR delivery["PDF or scan delivery"] --> ingest["ingest and classify\nai-classifypdf"] ingest --> profile["schema derivation\ndocument profile"] profile --> pilot["pilot gate\n5-page sample"] pilot --> ocr["schema-first OCR batch\nqwen3.8:27b-mlx"] ocr --> certify["certify-and-learn\npdai-audit --all"] certify --> learn["learning loop\nannotations A/B"] learn -. "regression gate" .-> ocr certify --> complete["corpus completion\nstanza · lemma · gazetteer · lexicon"] complete --> register["readiness manifest\nedition registry"] register --> viewer["ai-viewer endpoint"]
5 The Production Pipeline
From document delivery to the ai-viewer endpoint: every stage, engine, and data store
5.1 What this chapter covers
This chapter describes the single, authoritative production pipeline that takes an incoming PDF or scan and delivers a certified, queryable PDAI document to the ai-viewer endpoint. The pipeline is unified: one pilot-gated-ocr-pipeline Partitur (v1.1.0), one pdai-audit engine, one learning loop. The per-collection forking that characterised earlier iterations is deprecated; this is the canonical path.
Every arrow corresponds to a concrete command or service. Every node corresponds to a system component described below.
5.2 Stage 1: Ingest and classify
Command: ai-classifypdf ingest --collection <name> or ai-classifypdf prepare <collection> followed by ai-classifypdf run.
What happens: The PDF is harvested into the collection folder on PDAI-1, page images are extracted at 300 DPI, and the document is classified into one of the recognised document-class templates: printed-scholarly, special-script, handwritten-archival, or catalogue. The preparation step produces ocr_classification.json and collection_profile.yaml — the plan the OCR stage reads.
Engines: - fastapple backbone (Apple fastimage + Ollama vision LLM) for highspeed ingest; kimi-k2.7-code:cloud is the structuring model. - Classic ai-scholar process path (kimi-k2.7-code:cloud, per-page fallback gemini-3.8-flash) remains available via --classic.
Output: pages/page_NNN/ folder tree under /Volumes/PDAI-1/pdai-collections/<collection>/<doc>/, with one PNG per page and pdai-config.json recording the harvest provenance.
ImportantClassification precedes OCR
Never begin production OCR without a completed ocr_classification.json. The template, DTD, and model are all determined here; guessing them inline is not permitted.
5.3 Stage 2: Schema derivation
Command: ai-classifypdf prepare <collection> (also produced in Stage 1 for standard collections; for formalised editions, a dedicated schema-derive skill).
For formalised editions — catalogues, letter registers, ephemerides — the schema derivation reads the printed front matter of the volume and derives the record schema before any OCR. The derivation produces:
config/page.dtd— grammar for one OCR’d pageconfig/<record>.dtd— grammar for the assembled recordconfig/audit_rules.yaml— declarative cue and vocabulary rulesconfig/page.css— collection overlay on the shared book sheet: measures (--pdai-note-width,--pdai-marker-gutter) and typography; margin geometry is shared (Chapter 2)
The derivation is never retrospective: the DTD is written before the first production page is processed. Evidence grades: front-matter (authoritative), facsimile-only (provisional), unattested (do not invent).
5.4 Stage 3: Pilot gate
Command: python3 scripts/pilot_gate.py --vol <vol> or via the Partitur stage draw-pilot-sample + execute-pilot-ocr + audit-structural-gate.
Iron rule: For any document with more than 50 pages, the pilot gate is mandatory before batch OCR is unlocked. Five pages are drawn randomly, OCR’d under the assumed DTD, and audited in three passes:
- DTD validation — the XML must be well-formed and valid.
audit_rules.yaml— vocabulary, cue, and SCH-* checks.- Visual facsimile comparison — every VIS-* class checked against the actual scan.
If any blocker is raised, the improvement loop runs before batch unlock.
5.5 Stage 4: Schema-first OCR batch
Command: python3 scripts/<collection>_ocr_run.py --vol <vol> (or chain_<collection>.sh for multi-volume succession).
Primary model: qwen3.8:27b-mlx (Ollama local, M1 Ultra, think=false, image scale 0.5, num_ctx=8192, 1 worker — optimal on this hardware). Fallback: kimi-k2.7-code:cloud. Special-script / handwritten: kimi-k2.7-code:cloud directly.
Every page is processed against the collection config/page.dtd, rendered by ai-viewer at the collection’s dedicated port, and logged to a JSONL telemetry file under /Volumes/PDAI-1/tmp/<collection>-ocr/<vol>.jsonl. The live monitor is available at http://localhost:<port>/ocr-monitor?volume=<vol>.
Key parameters (qwen3.8:27b-mlx, validated Aug 2026):
| Parameter | Value | Effect |
|---|---|---|
num_batch |
512 | Prefill latency: 0.1 s (vs 26 s at default) |
num_ctx |
8192 | Sufficient for one PDAI page |
num_predict |
2048 | Avoids truncation on dense tables |
reasoning |
false | Required; thinking leaks in output otherwise |
| Workers | 1 | Optimal on M1 Ultra (2 workers do not increase throughput) |
5.6 Stage 5: Certify and learn
Command:
pdai-audit --collection <name> --vol <vol> --all --emit-annotationsThis is the certify-and-learn stage of Partitur pilot-gated-ocr-pipeline v1.1.0, registered at pipeline/ocr/pilot-gated in partitur_register.duckdb. It is not optional. A volume is not certified until this stage passes with 0 structural blockers and 0 visual blockers.
5.6.1 What --all runs
| Sub-stage | Flag | Output |
|---|---|---|
| Structural audit | (always) | quality/<vol>_defects.json |
| Visual layout scan | --visual |
quality/<vol>_visual_audit.json |
| Tag hierarchy protocol | --protocol |
quality/<vol>/tag_hierarchy_protocol.jsonl |
| Figure completion census | (part of --protocol) |
Embedded in protocol manifest |
| Fault dossier assembly | --dossier |
quality/<vol>/learning_fault_dossier.json |
| LettreAI Typst PDF | --typst |
quality/<vol>/audit_report_<vol>.pdf |
5.6.2 The tag hierarchy protocol
Every page in the volume is logged as a content-free structural fingerprint: tag names, nesting, and structural attributes only — no transcription content. Each page receives a SHA-256 hash of its pure S-expression. This creates the zero-regression anchor: when a prompt overlay or normalizer is updated, diff_tag_protocols() compares structural hashes in under 50 ms across all pages, catching any lost table, corrupted equation, or newly introduced element instantly.
5.6.3 Figure completion census
Every <figure> element is audited for:
@refattribute (required)<caption>element (printed caption, if present in the facsimile)<description>element (rich geometric or diagrammatic description)
Completion rates are expressed as four states: complete_full (caption + description), complete_desc (description only, standard when no printed caption exists), caption_only, and empty. Target: 100% of figures at complete_full or complete_desc.
5.6.4 The four-layer fault model
Every assembled fault case is attributed to one of four root-cause layers:
| Layer | System | Example fault |
|---|---|---|
| Layer 1: Prompt/Vision LLM | composed partitur: page language + <prefix>.base.j2 + engine overlay <prefix>.<engine>.j2 (qwen38, kimi, gemini38, qwen3vl) |
Missing <description> in figures; margin type or side wrong |
| Layer 2: Runner Normalizer | _clean_xml() in OCR runner |
Unjoined line-end hyphen before <br/> |
| Layer 3: Typographic CSS | pdai-page.css margins; config/page.css measures |
A marginal in the text column; notes overlapping |
| Layer 4: DTD/Record Schema | config/page.dtd |
<marginal> not declared in <footnote> |
5.6.5 The learning bridge (--emit-annotations)
Every assembled fault case is posted to the ai-query server (POST /api/annotations, port 5113) as an ocr-audit annotation, carrying: the fault’s collection, volume, and page coordinates; the observed defect (full FaultCase dataclass); the targeted fix directive; and the regression prevention anchor. The payload is also persisted as quality/<vol>/learning_annotations_payload.json.
Once annotations are registered, the A/B-gated instruction learning loop is triggered via POST /api/learn/from-annotations. The loop proposes a patch to the responsible SKILL.md (Layer 1–2) or schema file (Layer 4), runs the golden test set before and after, and promotes the patch only when the pass rate improves with no regression. This is an iron rule: no ad-hoc fixes to generation errors of a Partitur — all corrections go through the versioned learning system.
NoteRepairing a produced collection
When certification finds defects in a collection that is already produced, the repair follows the cure method: diagnosis, attribution to a layer, a remedy tested on 3–5 specimen pages, application to the flagged pages only. See Improving collections.
5.7 Stage 6: Corpus completion
Command: pdai-complete --collection <name>
After the volume is certified, the corpus completion pipeline runs the remaining indexing stages in order, checking for each stage’s readiness before proceeding:
| Stage | Engine | Output |
|---|---|---|
| OCR | (already done) | pages/page_NNN/*.xml |
| Structure | ai-scholar segment + materialise |
structure.xml, segments/ |
| Stanza ingest | ai_lemma_index stanza_ingest |
stz_sentences, stz_entities in corpus DuckDB |
| Lemma index | ai_lemma_index build_index |
lemma_index.parquet (postings) |
| Gazetteer | ai_lemma_index build_gazetteer |
Gazetteer entries in DuckDB |
| LWgrammar lexicon | ai_lemma_index lwgrammar_build |
<collection>_lwgrammar.duckdb |
| ChromaDB vectors | ai_lemma_index chroma_build |
Per-collection BGE-M3 vector store |
| Readiness manifest | pdai-complete status |
status.json on the collection |
5.7.1 Contextual data stores
After corpus completion, a PDAI collection maintains the following data stores that the query layer reads:
| Store | Location | Contents |
|---|---|---|
| Corpus DuckDB | /Volumes/PDAI-1/pdai-collections/<name>/<name>.duckdb |
Documents, letters, persons, places, dates, sentences |
| Lemma parquet | Same volume, lemma_index.parquet |
Inverted postings: lemma → (doc, pidx, tid) |
| Entity registry | entity_registry.duckdb |
Person/place aliases and canonical forms |
| Gazetteer | Embedded in corpus DuckDB | stz_entities; gazetteer expansions |
| LWgrammar lexicon | <collection>_lwgrammar.duckdb |
OED-form usage entries per lemma |
| Partitur register | development/ai-agent-orchestrator/partitur_register.duckdb |
Goal contracts, plan paths, stage registry |
| Audit vocabulary | config/audit_rules.yaml |
Cue rules, vocabulary, forbidden tokens |
5.8 Stage 7: Registration and the ai-viewer endpoint
Command: ae-platform port register --project <viewer> (already registered) or via the editions.yaml and edition registry.
When a volume passes pdai-complete and status.json reports ready, it is automatically visible in the ai-viewer instance for its collection. The edition registry maps collection names to viewer ports:
| Collection class | Viewer port | Instance |
|---|---|---|
| Kepler (prim/sec) | 5104 | kepler-gw edition |
| Pirckheimer | 5108 | pirckheimer edition |
| Astronomical Diaries | 5112 | diaries edition |
| Medieval Sec (Bischoff) | 5112 | Shared with diaries |
| Enestroem | 5100 | enestroem edition |
| Kármán archive | 5725 | karman edition (separate supervisor) |
| Editions landing | 5111 | Hub for all editions |
The viewer serves: - Facsimile + structured transcription in a dual-pane split view - Segment navigation via manifest.json - Collection-specific typographic rendering (page.css) - The <ai-query-chat> web component (Shadow DOM) for inline query access - The integrated per-collection /audit page for live quality scoring
5.9 The Partitur contract
The full pipeline is formalised as Partitur pilot-gated-ocr-pipeline v1.2.0, registered at goal path pipeline/ocr/pilot-gated in development/ai-agent-orchestrator/partitur_register.duckdb.
| Stage | Goal contract |
|---|---|
assess-document-scale |
total_pages > 50 → pilot_required == true |
profile-document |
every finding graded; page classes cover all pages |
derive-document-grammar |
DTD parses; no unattested vocabulary declared |
draw-pilot-sample |
len(sample_pages) == 5 and all pages unprocessed |
execute-pilot-ocr |
All pilot pages well-formed XML |
audit-structural-gate |
blockers == 0 |
train-layer |
retested_defects == 0 |
fit-check |
fit == pages and fails == [] |
select-model |
engine has a passed layer; a changed default is owner-confirmed |
unlock-and-execute-full |
done == total_pages, failed == 0 (threshold 0.99) |
certify-and-learn |
blockers == 0 and vis_blockers == 0; annotations_emitted == fault_cases |
The Partitur is bound through a setting file (skills/pipeline/pilot-gated-ocr/settings/<collection>-<vol>.partitur.setting.yaml) that provides the concrete collection, volume, primary_model, and dtd_path inputs before execution. The Conductor never executes an unbound Partitur.
5.10 Steering catalog
pdai/collections/ocr_catalog.yaml is the one file that maps every collection to its Partitur, model, ports, data roots, and DTD class. The pdai-audit CLI reads this catalog automatically so metadata (viewer port, model identities, edition label, volume titles) is always derived from the catalog rather than hardcoded in audit scripts.
WarningOne audit engine, one entry point
pdai-audit --collection <c> --vol <v> --all is the only supported way to run a comprehensive audit. The per-collection scripts/audit_check.py files are deprecated forwarding shims — they inject --collection <c> and call schema_audit.cli.main. They continue to work for backward compatibility but should not be maintained further.
pdai-audit is the console script of the schema-audit package at pipeline/schema-audit/. The corpus-readiness audit of ai-lemma-index (status.json) is a separate command, pdai-corpus-audit.