10 Audit Reference

SYN-, CHR-, SCH-, VIS- defect taxonomy, unified pdai-audit CLI, tag hierarchy protocol, figure census, fault dossier, and the learning bridge into ai-query

10.1 One audit engine

All OCR quality auditing for the PDAI workspace is owned by a single package:

pdai/pipeline/schema-audit/
  src/schema_audit/
    cli.py          — entry point: pdai-audit
    auditor.py      — structural + vocabulary checks
    visual_audit.py — VIS-* visual layout scan
    figure_audit.py — figure completion census
    tag_protocol.py — content-free tag hierarchy + SHA-256 hashing
    fault_dossier.py — fault case assembly + four-layer attribution
    typst_report.py  — LettreAI Typst PDF generation
    learn_bridge.py  — fault → ai-query annotation bridge

Entry point:

pdai-audit --collection <c> --vol <v> [--all] [--emit-annotations]

Metadata (edition label, model names, viewer port, template paths) is read automatically from collections/ocr_catalog.yaml and each collection’s config/volumes.yaml. No hardcoded collection-specific logic exists in the shared package.

10.1.1 Per-collection shims (deprecated)

The per-collection scripts/audit_check.py files are deprecated forwarding shims. They inject --collection <name> and forward every other argument to schema_audit.cli.main. They work for backward compatibility but must not be maintained further. Run pdai-audit directly.

10.2 CLI flags

Flag	Stage triggered	Output file
(none)	Structural audit only	`quality/<vol>_defects.json`
`--visual`	Visual layout scan	`quality/<vol>_visual_audit.json`
`--protocol`	Tag hierarchy protocol + figure census	`quality/<vol>/tag_hierarchy_protocol.jsonl`
`--dossier`	Fault dossier assembly	`quality/<vol>/learning_fault_dossier.json`
`--typst`	LettreAI Typst PDF	`quality/<vol>/audit_report_<vol>.pdf`
`--all`	All of the above	All output files
`--emit-annotations`	Bridge fault cases to ai-query	`quality/<vol>/learning_annotations_payload.json`

The PDAI cure uses these codes and the text-validity codes (TXT-*, FN-*, SEG-ORDER) as its diagnosis and assigns each code a layer; see Attribution.

10.3 Defect code namespace

All defect codes use the shared schema_audit package. The taxonomy has four namespaces:

Prefix	Layer	Examples
`SYN-*`	Structural / XML syntax	`SYN-MALFORMED`, `SYN-FENCE`, `SYN-THINK-LEAK`
`CHR-*`	Character / encoding	`CHR-FORBIDDEN`, `CHR-AMP-UNESCAPED`
`SCH-*`	Schema / DTD conformance	`SCH-REND-MISSING`, `SCH-HTML-TAG`, `SCH-UNKNOWN-ELEM`
`VIS-*`	Visual layout (CSS rendering)	`VIS-BR-IN-P`, `VIS-HYPHEN-SPLIT`, `VIS-TABLE-FLATTENED`

10.4 SYN codes

Code	Trigger	Severity
`SYN-MALFORMED`	XML is not well-formed	blocker
`SYN-DTD-FAIL`	XML fails DTD validation	blocker
`SYN-FENCE`	Markdown code fence in output	repair
`SYN-THINK-LEAK`	`<think>` or `</think>` in output	repair
`SYN-EMPTY`	Page contains no text content	warn

10.5 CHR codes

Code	Trigger	Severity
`CHR-FORBIDDEN`	Forbidden Unicode code point	blocker
`CHR-AMP-UNESCAPED`	Literal `&` in text content	repair
`CHR-ANGLE-UNESCAPED`	Literal `<` or `>` in text content	repair

10.6 SCH codes

Code	Trigger	Severity
`SCH-REND-MISSING`	`<hi>` without `rend` attribute	repair
`SCH-REND-UNKNOWN`	`rend` value not in closed vocabulary	repair
`SCH-HTML-TAG`	`<b>`, `<i>`, `<sup>`, `<sub>`, `<br>` in output	repair
`SCH-UNKNOWN-ELEM`	Element not declared in `page.dtd`	blocker
`SCH-CUE-MISSING`	Printed table cue without `<table>` element	blocker
`SCH-TABLE-FLAT`	Table rendered as consecutive `<p>`	blocker
`SCH-LEGACY-FLAT`	(diaries) flat text where `<tablet>` expected	blocker
`SCH-NUMBER-KIND`	(diaries) unrecognised line-number kind	warn
`SCH-SECTION-UNKNOWN`	(diaries) unknown section type	warn
`SCH-REF-MISSING`	(diaries) cross-reference without `@ref`	warn

10.7 VIS codes

Visual defects detected by visual_audit.py after comparing the CSS-rendered XML with the facsimile page image.

Code	Trigger	Root cause layer
`VIS-BR-IN-P`	Line break inside paragraph causes layout fragmentation	Prompt/normalizer
`VIS-HYPHEN-SPLIT`	Hyphenated word split across two `<p>` elements	Prompt
`VIS-ILLEGAL-INLINE`	Block element (`<equation>`, `<figure>`) inside `<p>`	Prompt
`VIS-TABLE-FLATTENED`	Table visible in facsimile but rendered as prose	Prompt
`VIS-MARGINAL-GUTTER`	Marginal note overflows into main text column	CSS

10.8 The tag hierarchy protocol

The tag hierarchy protocol provides a content-free structural fingerprint of each page for sub-50 ms regression testing after OCR improvements.

What is stored: For each page, the protocol logs the S-expression tree of XML tag names, nesting order, and structural attributes only — no transcription text. A SHA-256 hash of the S-expression serves as the page’s structural identity.

How regression testing works:

from schema_audit.tag_protocol import diff_tag_protocols
deltas = diff_tag_protocols(protocol_v1, protocol_v2)
# Returns per-page structural changes: gained/lost elements, depth shifts

A prompt overlay update that inadvertently drops all <table> elements will surface within 50 ms as a mass structural delta on table pages. A fix that correctly joins hyphenated words across <p> boundaries shows no structural change (the same elements remain, differently populated).

The protocol JSONL file is:

quality/<vol>/tag_hierarchy_protocol.jsonl

One JSON object per page, with fields pidx, sha256, depth_histogram, element_counts, and s_expr_truncated (first 512 chars of the S-expression).

10.9 Figure completion census

Every <figure> element in the volume is audited for three completion components:

Component	XML	Meaning
`@ref`	Required attribute	Links figure to extracted image file
`<caption>`	Child element	Transcribed printed caption
`<description>`	Child element	Rich description of the figure content

Completion is reported in four states:

State	Meaning
`complete_full`	Both `<caption>` and `<description>` present
`complete_desc`	`<description>` only (standard when no printed caption)
`caption_only`	`<caption>` but no `<description>` — a repair target
`empty`	Neither element present — a blocker in the Typst report

Target: 100% of figures at complete_full or complete_desc. Any empty figure is registered as a fault case in the dossier.

10.10 The fault dossier

The fault dossier (learning_fault_dossier.json) is the machine-readable artefact that feeds the OCR learning loop. It assembles all non-trivial defects into structured FaultCase records.

Each FaultCase contains:

@dataclass
class FaultCase:
    fault_id: str          # e.g. "FAULT-kgw15-p042-001"
    collection: str
    volume: str
    page: str              # pidx
    defect_code: str       # SYN-*, SCH-*, VIS-*, or composite
    fault_summary: str     # one sentence
    fault_description: str # precise description for the learning loop
    root_cause_layer: int  # 1=Prompt, 2=Normalizer, 3=CSS, 4=DTD
    root_cause_detail: str
    targeted_fix: dict     # {target_file, rule, regression_anchor}
    regression_anchors: list[str]  # pidx values that must not regress
    severity: str          # "blocker", "repair", or "warn"
    xml_evidence: str      # the offending XML fragment

The targeted fix names the file to patch and the rule to add or amend. The regression anchors are page indices that the fix must not break. Both fields are consumed by the learning loop when it proposes an instruction patch.

10.10.1 Four root-cause layers

Layer	System	Typical fix target
1 — Prompt/Vision LLM	Jinja2 overlay	`config/<collection>-ocr.qwen38.j2`
2 — Runner Normalizer	Python normaliser	`_clean_xml()` in OCR runner
3 — Typographic CSS	Stylesheet	`config/page.css`
4 — DTD/Record Schema	Grammar file	`config/page.dtd`

A fault cannot be fixed by patching the wrong layer: a CSS rendering artefact fixed by a prompt patch will reappear on new pages; a prompt omission patched in CSS will not fix historical pages. The layer attribution is the learning loop’s routing signal.

10.11 The learning bridge (`--emit-annotations`)

When --emit-annotations is passed to pdai-audit, every FaultCase in the dossier is posted to the ai-query server as an ocr-audit annotation:

POST http://127.0.0.1:5113/api/annotations
Content-Type: application/json

{
  "phase": "ocr-audit",
  "subagent": "ocr:kepler-prim",
  "raw_query": "kepler-prim/kgw-15/p042",
  "observed": { ...FaultCase fields... },
  "comment": "[FAULT-kgw15-p042-001] Table flattened to <p> on p042 ...",
  "expected": "SCH-TABLE-FLAT rule: require <table> when | cue present",
  "instruction_ref": "config/kepler-ocr.qwen38.j2",
  "status": "open"
}

The payload is also persisted as quality/<vol>/learning_annotations_payload.json for traceability and offline re-ingestion when the server is unreachable.

10.11.1 What happens after posting

The ai-query annotation loop (port 5113, assessment.duckdb) receives the annotation and queues it for the A/B-gated learning pass:

POST /api/learn/from-annotations proposes a patch to the SKILL.md or schema file named in instruction_ref.
The patched instruction is stored as a new version in instruction_versions with status candidate.
The golden test suite is re-run with the patched instruction.
If the golden pass rate improves with no regression on the regression_anchors pages, the version is promoted to active.
If it does not improve, the candidate is rejected and kept in the registry for forensic purposes.

Iron rule: Every OCR defect that reaches the fault dossier must enter the annotation system and be addressed through the versioned learning loop. No ad-hoc fix to a prompt or normalizer is permitted outside this flow. Code-side probes are supporting signal only; the responsible SKILL.md is the patch target.

10.12 `audit_rules.yaml` structure

Each collection maintains a config/audit_rules.yaml consumed by the shared Auditor class. The file has four sections:

syntax:
  strip_think: true      # <think> blocks → SYN-THINK-LEAK
  strip_fences: true     # ``` fences → SYN-FENCE
  validate_dtd: true     # DTD validation → SYN-DTD-FAIL

forbidden_tokens:
  - pattern: "<b>"
    code: SCH-HTML-TAG
    severity: repair
  - pattern: "&(?!amp;|lt;|gt;|apos;|quot;)"
    code: CHR-AMP-UNESCAPED
    severity: repair

vocabulary_rules:
  - element: hi
    attribute: rend
    allowed: [italic, bold, spaced, smallcaps, superscript, subscript]
    code: SCH-REND-UNKNOWN
    severity: repair
  - element: marginal            # the page-language vocabulary, read from the DTD
    attribute: type
    allowed: dtd                 # pdai-core.mod + the collection's marginal-type.extra
    require_on: partitur         # a missing @type is a defect on pages with <meta partitur>
    code: SCH-MARGINAL-TYPE
    severity: repair

cue_rules:
  - pattern: "\\|"        # pipe character often signals a table
    required_element: table
    code: SCH-CUE-MISSING
    severity: blocker

10.13 Negative control requirement

Every audit_rules.yaml must be validated with a negative control test suite (test_audit_rules.py) that:

Injects each defect type deliberately into a known-good XML sample.
Asserts that the expected code is raised.
Asserts that no code is raised for the clean control.

python3 collections/<name>/scripts/test_audit_rules.py
# Expected: 10/10 PASS

A rule that does not trigger on a deliberately injected defect is not a rule.

10.14 Running audits

# Full audit for any collection volume, all stages
pdai-audit --collection kepler-prim --vol kgw-15 --all --emit-annotations

# Structural audit only
pdai-audit --collection medieval-sec --vol Bischoff1998

# Structural + visual without Typst report
pdai-audit --collection diaries --vol adrt1 --visual --protocol --dossier

# Legacy (deprecated shim — still works)
python3 collections/kepler-prim/scripts/audit_check.py check --vol kgw-13

10.15 Composite scoring in the learning loop

The AuditEvaluator (development/ai-agent-orchestrator/src/orchestrator/learning/audit_evaluator.py) blends objective defect counts into the learning loop grade:

composite = llm_score × (1 − penalty)
penalty   = blocker_fraction × 0.5 + repair_fraction × 0.2

A promoted overlay must reduce measured blockers and repairs compared to the baseline, not merely win a self-assessment. The regression_anchors from the fault dossier are the exact pages that must show 0 blockers after promotion.

10.16 Naming collision note

“Audit” appears in two places in the PDAI workspace:

pdai-audit (this document): the OCR quality engine in pipeline/schema-audit/ — structural, visual, figure, dossier, report, bridge.
pdai-complete status: the corpus-readiness audit in ai-lemma-index — checks whether OCR, stanza, lemma, gazetteer, and Chroma stages are complete.

They are orthogonal. pdai-audit is about transcription quality; pdai-complete is about corpus stage completion. Both write a status.json into their respective domains (quality/ and the collection root on PDAI-1).