19 Reference

Metrics, thresholds, models, commands and the schema package

19.1 Metrics and thresholds

19.1.1 Page verdict (text validity)

pdai-validity (check version text-validity/1.10) compares the Apple Vision reading of the scan with the text of the page XML. The verdict of a page follows from its flags:

Verdict	Condition
`fail`	at least one severe flag: `TXT-PAGE-LOST`, `FN-MISSING`, `TXT-EXCESS`, `BODY-IN-FN`, `FN-IN-BODY`
`warn`	flags, none of them severe (e.g. `TXT-GAP`, `FN-PARTIAL`, `SEG-ORDER`)
`ok`	no flag
`unverifiable`	mean Vision confidence below 0.45, or the collection class is not readable by Vision (`handwritten-archival`, `special-script`)

Detector parameters: a scan line counts as missing below a token coverage of 0.5; two consecutive missing body lines with confidence ≥ 0.5 make a TXT-GAP; fewer than 12 scan tokens mark a (near) blank page; TXT-EXCESS requires more than 1.8 × the scan tokens + 40 and a precision below 0.6.

19.1.2 Collection verdict

Verdict	Condition (n = pages)
`valid`	fail rate ≤ 1 %
`minor-defects`	fail rate ≤ 5 %
`defective`	fail rate > 5 %
`unverifiable`	more than half of the pages `unverifiable`, or class not readable

19.1.3 Iteration metrics

Metric	Source	Condition in the cure
fail count	`text_validity.json`	not higher after the iteration (certify)
structural defect count per code	`pdai-audit` defect register	not higher (certify)
new defect class	diagnosis vs. certification	none (certify)
blockers on specimen pages	`pdai-audit`	0 (gate)
composite score of an overlay	`cure_learn`: mean inner score × (1 − audit penalty)	candidate above baseline (learn)

The inner score of a page in cure_learn is 0 with any audit blocker, otherwise the validity verdict mapped to ok 1, warn 0.75, unverifiable 0.5, fail 0. The audit penalty is blocker_fraction × 0.5 + repair_fraction × 0.2.

19.2 Model policy

The roles are defined in pdai/skills/pipeline/pdai-cure/models.yaml and resolved by ocr_runner.cure_models.resolve_role. A collection overrides a role with models: {<role>: <model>} in its config/cure.yaml.

Policy

No LLM where a deterministic check exists (scope, validity and audit checks, gates, normalizer, certification, downstream).
The default of every LLM role, text and vision, is kimi-k2.7-code:cloud via Ollama Cloud (owner decision 2026-09-26).
gemini-3.8-flash is only the fallback of the Kimi default. It runs when Kimi fails on the item: an HTTP error, a timeout, an empty answer, or an answer that fails the step’s check. With the environment variable PDAI_CURE_FALLBACK=off a run has no fallback: a failed Kimi item is recorded as an error and no Gemini call is made.
Local qwen3.8:27b-mlx is the route for harvest and for runs that must stay local.
Every A/B run is recorded in model_evidence.jsonl. No untested model is adopted.

Role	Stage	Kind	Default	Fallback
`reocr`	apply	vision	head of the catalog `engine_chain` (`kimi-k2.7-code:cloud`, then `qwen3.8:27b-mlx`)	`gemini-3.8-flash`, only after every chain engine failed on the page
`print-first-pass`	apply	vision	`kimi-k2.7-code:cloud` (local: `qwen3.8:27b-mlx`)	`gemini-3.8-flash`
`validity-judge`	diagnose	vision	head of the catalog chain	`gemini-3.8-flash`
`attribute-uncoded`	attribute	text	`kimi-k2.7-code:cloud`	`gemini-3.8-flash`
`overlay-propose`	remedy	text	`kimi-k2.7-code:cloud`	`gemini-3.8-flash`
`learning-text`	learn	text	`kimi-k2.7-code:cloud`	`gemini-3.8-flash`
`schema-derive`	remedy	text	`kimi-k2.7-code:cloud`	`gemini-3.8-flash`
`facsimile-check`	certify	vision	`kimi-k2.7-code:cloud` (candidate: `qwen3.8:27b-mlx` first tier)	`gemini-3.8-flash`
`css-classify`	certify	text	`kimi-k2.7-code:cloud`	`gemini-3.8-flash`
`downstream-nlp`	downstream	local	Stanza + GLiNER2.5 + BGE-M3	—

Text roles run through ocr_runner.cure_models.runtime_for_role, which calls the default and hands the item to the fallback on an exception, an empty answer or an answer that the step’s check rejects; the route taken is recorded with the result (author_route in the cure_learn summary).

Ollama call options from live probes: kimi-k2.7-code:cloud, deepseek-v4.1-flash:cloud and qwen3.8:27b-mlx run with think: false.

19.2.1 Retired and not adopted models

Model	Status	Reason
`qwen3.5:cloud`	retired	HTTP 410: Ollama withdrew `qwen3.5:397b` on 2026-09-25 (probe 2026-09-26)
`kimi-k3:cloud`	retired	HTTP 402 without extra-usage credits
`deepseek-v4-flash:cloud`	not adopted	text only, HTTP 400 on images
`glm-5.3:cloud`	not adopted	overlay-propose A/B: candidate 0.714 against baseline 0.855, one new blocker
`gemma4:31b-cloud`	not adopted	untested on page OCR
`qwen3-vl:235b-cloud`	not adopted	untested on page OCR

resolve_role refuses a model outside the Gemini allowlist, a retired model and a not adopted model.

19.2.2 Model evidence

Each run of ocr_runner.cure_learn appends one record to pdai/skills/pipeline/pdai-cure/model_evidence.jsonl. The A/B of the role overlay-propose on 2026-09-26 revised the overlay ocr-overlay:copernicus-prim:kimi (engine kimi-k2.7-code:cloud) for one round with each author:

Sample	Author	Baseline	Candidate	Duration
3 pages	`glm-5.3:cloud`	1.000	1.000	155 s
3 pages	`gemini-3.8-flash`	1.000	1.000	66 s
7 pages	`glm-5.3:cloud`	0.855	0.714	292 s
7 pages	`gemini-3.8-flash`	0.857	0.857	171 s
7 pages	`kimi-k2.7-code:cloud`	0.857	0.857	206 s

The 3-page sample does not separate the authors, because the baseline already scores 1.000. On 7 pages the candidate of glm-5.3:cloud introduced one blocker (GW-IV page 19: XML not well-formed, page text lost); the candidates of gemini-3.8-flash and kimi-k2.7-code:cloud left every page verdict unchanged. No candidate solved the FN-MISSING of Copernicus1873 page 50. The baseline runs differ by at most 0.0017, so the drop of the glm candidate exceeds the run-to-run variation, while Kimi and Gemini are equal on this sample. One round on one sample is weak evidence; the Kimi default rests on the owner decision, the evidence only excludes glm-5.3:cloud.

The two-tier setup of facsimile-check needs visual-audit specimens with adjudicated discrepancies before its A/B can run; none are registered yet.

19.3 Commands

All commands run from ~/ai-mac-studio/pdai. The ocr-runner modules run with the orchestrator venv (~/ai-mac-studio/development/ai-agent-orchestrator/.venv/bin/python -m …).

Stage	Command	Writes
diagnose	`pdai-validity --collection C [--doc D]`	`text_validity.json`, `pages.jsonl`
diagnose	`pdai-validity --worklist C`	`<data>/quality/text_validity/reocr/<doc>.json`
diagnose	`pdai-audit --collection C --vol D --dossier --protocol --quiet`	dossier, tag protocol
certify	`pdai-audit --collection C --vol D --all`	+ Typst report `audit_report_<vol>.pdf`
certify	`pdai-coherence --collection C --vol D`	CSS coherence report
certify	`pdai-validity --adjudicate C --page P --codes X --evidence E --by B`	adjudication register
gate	`python -m ocr_runner.normalize_apply --collection C --doc D --specimen P1,P2,P3`	`<data>/quality/<doc>_norm_gate.json`
apply	`python -m ocr_runner.normalize_apply --collection C --doc D --apply`	page XML (gated normalizer only)
gate	`python -m ocr_runner.cure_run --collection C --doc D --pilot --also P… --sample N --targeted 0`	pilot tree under the collection tmp directory
apply	`python -m ocr_runner.cure_run --collection C --doc D --worklist --max-pages N --max-minutes M`	page XML of flagged pages, backups `page_NNN.xml.vN`
remedy	`python -m ocr_runner.cure_learn --collection C --engine E --rounds R [--author M] [--apply]`	version tree, `model_evidence.jsonl`; overlay only with `--apply`
setup	`python -m ocr_runner.cure_settings --collection C [--force]`	document settings from `setting_defaults`
downstream	`pdai-complete --collection C [--dry-run]`	corpus stages of changed pages

--dry-run of cure_run lists the pages it would process without calling an engine.

The Partitur runs through the Conductor:

cd ~/ai-mac-studio/development/ai-agent-orchestrator
.venv/bin/orchestrator run ~/ai-mac-studio/pdai/skills/pipeline/pdai-cure/partitur.yaml \
    --setting cure-copernicus-prim [--input-file iteration.yaml]

The input file carries remedy, until and apply for one iteration. Setting ids are global in partitur_register.duckdb; the Settings of pdai-cure are named cure-<collection>.

19.4 The schema package of a collection

A collection is ready for the cure when its code folder pdai/collections/<c>/ holds:

File	Content
`config/cure.yaml`	instruction prefix, audit rules path, `placement_rule`, `reject_codes`, guard options, normalizer module, `setting_defaults`, `doc_notes`, budget, model overrides
`config/page.dtd`	page grammar the audit validates against
`config/audit_rules.yaml`	structure rules; the rule named by `placement_rule` carries the note placement map per document; `layer` per rule overrides `defect_layers.yaml`
`config/volumes.yaml`	document metadata for `pdai-audit`
`config/page.css`	measures and typography over the shared sheet (margin geometry is shared, Chapter 2); the viewer reads the deployed copy `<data>/page.css`, written by `python -m ocr_runner.page_language deploy`
`ai-instructions/<prefix>.base.j2`	shared OCR instruction
`ai-instructions/<prefix>.<engine>.j2`	one overlay per engine key; keys from the one table at the top of `ocr_catalog.yaml`; composed by `cure_config.compose` after the page-language fragment; `pdai-partitur check` lists missing overlays
`ai-instructions/settings/<doc>.partitur.setting.yaml`	bound values per document (engine chain, image scale, notes)

and when collections/ocr_catalog.yaml has an entry with data path, tmp directory, viewer port and engine_chain. The cure’s stage scope lists every missing part as a gap. The data themselves stay on /Volumes/PDAI-1/pdai-collections/<c>/.