Famulus reference
Reference data sources with addresses, citations and traces for every item
A famulus reference is a reference data source in need of famulus proof. It delivers grounding data for every AI request with low latency and with full references to each single item used: a citation in scientific form, a provenance chain, a validation grade and a lake snapshot. Each answer that uses it can later be justified item by item.
Every PDAI collection becomes a famulus reference by the intake skill (new collections) or by migration (existing collections). The data is stored in the DuckLake and served by the famulus-reference MCP server (port 5135). Component overview: The Famulus universe.
| Package | pdai/famulus-reference/ (README is authoritative) |
| Schema | famulus-reference/1 (src/famulus_reference/schema/core.sql, lake.sql) |
| Server | MCP streamable HTTP http://127.0.0.1:5135/mcp; tailnet http://‹mac-im-tailnet›:5135/mcp |
| Registry | platform/registry/mcp/famulus-reference/mcp.yaml (proxied_via: ai-platform-mcp) |
| Route | /famulus-reference in platform/config/routes.yaml (deploy: planned) |
The fr: address
fr:<collection>/<doc>[/<pidx:04d>[/<segment>]]| Part | Content | Example |
|---|---|---|
<collection> |
collection id = lake schema name | hartmann |
<doc> |
document folder name | bsb00028171 |
<pidx> |
page index, four digits, zero-padded | 0137 |
<segment> |
stable segment label, or s + 8 hex of sha256 over (doc, pidx, bbox, type) |
print1, s3fa09c1e |
The level of an item follows from the number of parts:
| Address | Level | Table |
|---|---|---|
fr:hartmann |
collection | fr_collection |
fr:hartmann/bsb00028171 |
document | fr_document |
fr:hartmann/bsb00028171/0137 |
page | fr_page |
fr:hartmann/bsb00028171/0137/print1 |
segment | fr_segment |
Use in famulus-proof. The address string is used verbatim as Clause.address. famulus_proof.lake.resolve_fr(address) returns the item, its text layers, the citation (own or nearest ancestor), the provenance and the snapshot; famulus_proof.lake.beleg(address) returns the clause fields (kind: beleg, label, address, excerpt). An unknown address returns found: false with the gap, never an empty pass.
Global identifiers. URN, ARK, DOI, GND, shelfmark and IIIF identifiers are stored in fr_identifier (scheme, value, resolver), not in the address. Example: fr:sundial/Hartmann1500 is the earlier copy of the Hartmann volume in the sundial collection; after the pilot, sundial/status.json marks it superseded_by: hartmann, and an alias row in fr_identifier connects it to fr:hartmann/bsb00028171.
What every item carries
| Property | Table | Content |
|---|---|---|
| Citation | fr_citation |
BibTeX, CSL-JSON, text citation, locator, gaps; derived only from biblio.xml or the collection descriptor |
| Provenance | fr_provenance, fr_source |
pipeline step, tool, model, instruction, input/output sha256, file, run; origin: holding institution, shelfmark, origin URL, IIIF manifest, licence, sha256 |
| Validation grade | fr_validation |
check, grade DOCUMENTED / INFERRED / HYPOTHESIS / GAP, famulus-raum/1 certificate sha256, audit rule |
| Identifiers | fr_identifier |
scheme, value, resolver |
| Snapshot | lake snapshot id | returned with every API answer (snapshot_id, snapshot_time, schema_version) |
| Harvest state | fr_harvest |
datestamp, set, deleted flag (OAI-PMH) |
Rule: citations are never invented
A citation is composed only from recorded bibliographic data. A required field that is not recorded is listed in gaps and is not filled in.
| Case | fr_cite result |
|---|---|
| item has its own citation | status: ok, with locator |
| only an ancestor (page, document, collection) has a citation | status: ancestor, cited_as: <ancestor address>, note to cite with that entry and the item address as locator |
| no recorded data for the item or any ancestor | status: gap, gaps: ["citation"], no citation composed |
Required fields per entry type: book (author or editor, title, year, publisher), article (author, title, journal, year), incollection (author, title, booktitle, year), letter (author, title, year), misc (title).
Sample BibTeX entry
Output of fr_bibtex(["fr:hartmann/bsb00028171/0137/print1"]). Field values are taken from the recorded BSB catalogue data (scholarly/research/hartmann/SOURCES.md); the item title and snapshot id are example values. No creator or year is recorded for the print, so the answer lists them as gaps instead of assuming an attribution.
@inbook{hartmannbsb000281710137print190c5,
title = {Print 1, scan 137},
booktitle = {[Collectio figurarum]},
urn = {urn:nbn:de:bvb:12-bsb00028171-2},
note = {Bayerische Staatsbibliothek München, Rar. 434},
addendum = {scan 0137, print1},
url = {https://www.digitale-sammlungen.de/de/view/bsb00028171},
famulusref = {fr:hartmann/bsb00028171/0137/print1 @snapshot 42}
}| Field | Rule |
|---|---|
| key | deterministic: alphanumeric tail of the address (max. 40 characters) + 4 hex of sha1 of the address |
famulusref |
the fr: address and the snapshot id; BibLaTeX ignores unknown fields, so the entry stays valid |
pages / addendum |
locator: numeric locator → pages, otherwise addendum |
title / booktitle |
item-level entry: item title as title, document title as booktitle (entry type book becomes inbook) |
Universal API
The same functions are available as MCP tools, as CLI subcommands (famulus-reference cite …) and as Python functions (famulus_reference.api).
| MCP tool | Arguments | Returns |
|---|---|---|
fr_list_collections |
— | registered collections with export time and row counts |
fr_describe_collection |
collection |
metadata, licence, rights holder, data source origin, identifiers, DTDs, counts |
fr_get_dtd |
collection, name? |
DTD text, version, sha256, status (draft / deployed / approved) |
fr_get_structure |
collection, doc? |
documents and their page and segment counts |
fr_list_segments |
collection, seg_type?, doc?, limit, offset |
segments with type, parent, page range, bbox, DTD element |
fr_get_item |
fr_id |
item row, text layers with sha256, identifiers |
fr_get_sidecars |
collection |
available extension and legacy tables |
fr_get_provenance |
fr_id |
provenance chain of the item and its ancestors, source origin |
fr_cite |
fr_id, style (all, bibtex, csl, text) |
citation, status, cited_as, locator, gaps, snapshot |
fr_bibtex |
fr_ids[] |
BibTeX entries (deduplicated), addresses without citation listed as gaps, snapshot |
fr_trace |
fr_id |
justification record: item, text sha256, provenance, validation records, aggregated evidence grade, citation, identifiers, snapshot |
fr_ground |
query, collection?, k, doc? |
lexical hits: fr: address, text layer, score, snippet, citation |
fr_sql |
sql, limit |
read-only result (one SELECT/WITH/SHOW/DESCRIBE; AST guard; no file readers; max. 500 rows, truncated flag) |
fr_snapshot_info |
— | current snapshot id, time, schema version |
fr_trace_run |
run_id |
justification of a past answer: tool, request, steps, SQL, fr: addresses, snapshot |
Justification by run_id
fr_cite, fr_bibtex, fr_trace, fr_ground and fr_sql record a run in the trace DB (famulus_lake_traces, _lake/famulus_reference_traces.duckdb, written only by the MCP server) and return its run_id.
| Table | Columns |
|---|---|
fr_run |
run_id, ts, client, tool, request (JSON), snapshot_id |
fr_run_step |
run_id, step, sql, fr_ids[], n_rows, ms |
A later validation or justification request (“which items supported this answer, in which state?”) is answered by fr_trace_run(run_id): it returns the addresses used and the snapshot at which they were read. With the snapshot, the items can be read again in exactly that state.
Reproducibility: snapshots and releases
| Mechanism | Effect |
|---|---|
| Snapshot | every lake write (one export = one snapshot) creates a snapshot id; readers can query any retained snapshot (time travel) |
| Cited snapshot | snapshots handed out by fr_cite / fr_bibtex (recorded in fr_run) are never expired |
| Release | main.fr_release(snapshot_id, tag, reason, fr_ids[], recorded_at): snapshots referenced by citations, landing manifests or famulus-proof certificates; never expired |
| Retention | other snapshots expire after 7 days (see DuckLake maintenance) |
| Read replica | readers attach famulus.read.ducklake, published after each write session; a running export never blocks a reader |
Distributed reference networks
The API does not depend on the PDAI collections: any institution or machine can run the same server over its own DuckLake with the same schema famulus-reference/1. A node adds collections by exporting them into its lake; the network is extended by adding nodes.
| Element | State |
|---|---|
| Same API per node | implemented (package famulus-reference, environment overrides FAMULUS_LAKE_CATALOG, FAMULUS_REFERENCE_BASE, FAMULUS_REFERENCE_PORT) |
| Federation registry | GET /peers serves platform/config/famulus_reference_peers.yaml (id, institution, MCP URL, OAI URL, collections, contact); currently peers: [] (stub) |
| Cross-peer union query | not implemented; harvesters follow each node’s OAI endpoint |
Harvesting by library, archive and data-store catalogs
| Route | Standard | Content |
|---|---|---|
GET\|POST /oai |
OAI-PMH 2.0, all six verbs | metadata formats oai_dc, oai_datacite; sets = collections; datestamps and deletions from fr_harvest; responses validated against OAI-PMH.xsd and oai_dc.xsd in tests |
GET /jsonld/<coll> |
schema.org Dataset + DCAT |
collection description for data catalogs and search engines |
GET /iiif/<coll>/<doc>/manifest.json |
IIIF Presentation 3 | canvases pointing at the holding institution’s image service |
GET /peers |
JSON | federation registry |
GET /health |
JSON | server state |
Example requests:
curl 'http://127.0.0.1:5135/oai?verb=ListRecords&metadataPrefix=oai_dc&set=hartmann'
curl 'http://127.0.0.1:5135/jsonld/hartmann'
curl 'http://127.0.0.1:5135/iiif/hartmann/bsb00028171/manifest.json'Implemented and stub
| Function | State |
|---|---|
fr: address, schema famulus-reference/1, per-collection famulus DB, lake export |
implemented |
MCP tools fr_*, trace DB, fr_trace_run |
implemented |
OAI-PMH (oai_dc, oai_datacite), JSON-LD/DCAT, IIIF Presentation 3 manifests |
implemented |
famulus-proof fr: resolver, famulus-notebook guard check of fr: citations |
implemented |
Deployment on mac-studio (launchd com.ai-platform.famulus-reference-mcp) |
prepared (deploy/install_studio.sh); route deploy: planned |
/peers federation |
stub (empty registry) |
| local IIIF Image API server | documented stub; manifests use the institution’s image service |
| DOI minting | documented stub |
| ResourceSync | documented stub |
| cross-peer union query | documented stub |
Usage examples
cd ~/ai-mac-studio/pdai/famulus-reference # clients: ~/ai-openscience/pdai/famulus-reference
.venv/bin/famulus-reference cite fr:hartmann/bsb00028171/0137/print1
.venv/bin/famulus-reference trace fr:hartmann/bsb00028171/0137
.venv/bin/famulus-reference ground "horologium" --collection hartmannVia ai-platform-mcp:
call_tool("famulus-reference", "fr_cite", {"fr_id": "fr:hartmann/bsb00028171/0137/print1", "style": "bibtex"})
call_tool("famulus-reference", "fr_trace_run", {"run_id": "<run_id from an earlier answer>"})