13  Diagnosis

Stages 0 and 1: scope and measurement

13.1 Stage 0 — Scope

The scope fixes what an iteration may touch. It reads three sources:

  • pdai/collections/ocr_catalog.yaml — data root on PDAI-1, tmp directory, viewer port and engine_chain of the collection;
  • pdai/collections/<c>/config/cure.yaml — what the catalog does not hold (instructions, rules, engine keys, reject codes, normalizer module, budget);
  • <data>/status.json — the corpus readiness written by pdai-complete.

It then checks the schema package: config/cure.yaml, config/page.dtd, config/audit_rules.yaml, config/volumes.yaml, page.css and the OCR templates in ai-instructions/. A missing data directory fails the stage. Missing package files do not stop a diagnosis; they are listed as gaps and mark the collection as not ready for repair.

cd ~/ai-mac-studio/development/ai-agent-orchestrator
.venv/bin/python -c "from ocr_runner.cure_config import load_cure, describe; \
print(describe(load_cure('copernicus-prim')))"

13.2 Stage 1 — Diagnose

Two measurements run per scope. Both are deterministic.

13.2.1 Text validity

pdai-validity --collection <c>              # all documents
pdai-validity --collection <c> --doc <d>    # merged into the stored report
pdai-validity --report <c>
pdai-validity --worklist <c>

pdai-validity (check version text-validity/1.10) reads every scan with Apple Vision and compares the reading with the page XML, zone by zone (header, main, footnotes, marginalia). Vision results are cached per image modification time; a rerun compares only the XML again and takes seconds for the 3,644 pages of copernicus-prim.

Each page receives one verdict:

Verdict Meaning Consequence
ok the XML covers the scan text never processed again
warn a partial defect, for example FN-PARTIAL listed for review
fail text lost or misplaced on the re-OCR worklist
unverifiable Vision confidence below 0.45 facsimile review only
Warningwarn can hide text loss

In copernicus-prim a footnote of Copernicus1873 p. 100 had lost two of its three paragraphs, and the page was reported as TXT-GAP of the main zone, because Vision had not separated the note zone. Before a class of warn pages is accepted, one specimen is compared with the facsimile.

Outputs:

File Content
<data>/text_validity.json verdict counts, fail rate, overall grade, flags per document and model
<data>/quality/text_validity/<d>.pages.jsonl verdict, flags, recall, precision and producing model per page
<data>/quality/text_validity/reocr/<d>.json the worklist: failing pages with the next engine to try

The overall grade is valid at a fail rate up to 1 %, minor-defects up to 5 %, defective above, and unverifiable when more than half of the pages cannot be read.

13.2.2 Structural audit

pdai-audit --collection <c> --vol <d> --dossier --protocol --quiet

The audit checks each page against page.dtd and the structure rules of audit_rules.yaml and writes, in pdai/collections/<c>/quality/<d>/:

  • learning_fault_dossier.json — fault cases with codes, severity, layer, targeted fix and regression anchor;
  • tag_hierarchy_protocol.json — the content-free tag structure of every page as an S-expression with its hash; the certification compares it with the protocol after the repair.

It also prints the figure census. On GW-IV it runs in 2.4 seconds.

13.2.3 Further measurements

  • Visual audit on the collection’s viewer port: facsimile next to the rendering, every flaw assigned to a layer (skill visual-layout-audit).
  • Corpus audit pdai-corpus-audit --collection <c>: sentence counts per language, coverage and staleness of the corpus stages.

13.3 In the Partitur

Stages scope and diagnose (ops pdai:cure_scope, pdai:cure_diagnose). The diagnosis op fails when pdai-validity or an audit exits with an error, or when the validity report does not cover the scope.