README

Folio stores normalized/enriched dataset JSONL as portable SQLite archive files. It is the database, archive, and browsing layer for data that has already been normalized by dataset/import tooling.

FolioBuilder turns explicit JSONL inputs into a standalone folio SQLite file. The owning application selects inputs and manages its workflow. Consumers such as zm can use the Symfony/DataContracts stack when present, while Python/R/SQLite users can query the archive directly.

Required: survos/field-bundle, survos/data-contracts. Suggested for ingest/write workflows: survos/jsonl-bundle, survos/import-bundle.

The application must explicitly register Survos\DataContracts\Path\DataPaths; FolioBundle never supplies a fallback. See docs/configuration.md for that service and the required multi-connection Doctrine setup. See docs/archive-metadata.md for the standalone archive metadata contract. See docs/publisher-registration.md for how a publishing app confirms and withdraws folios with Harvest (PublisherRegistrar). See docs/presentation-layer.md for the proposal to use folios as narrative institutional presentation packages.

Building and command cleanup

See Building folios for the service API and supported commands. Dataset-specific build, archive, translate and ingest CLI shortcuts are removed; no replacement shortcuts are installed in another bundle.

Archive Contract

A folio file stores canonical rows in item and self-describing metadata alongside them:

schema_table and schema_property describe observed DTO types and fields in this archive.
schema_property.stats stores field profile output from survos/jsonl-bundle's profiler.
docs stores generated JSON/Markdown documentation for humans, report writers, and AI agents.
generated dto_* SQLite views project JSON fields into query-friendly columns.
term_set and term store standalone controlled vocabularies and facets.

The metadata snapshot describes actual observed data, not the entire DTO contract universe. DTO classes from survos/data-contracts annotate observed fields with labels/descriptions when available, but consumers do not need PHP code to understand an archived folio.

Search and Publication Notes

FolioBuilder loads rows, snapshots observed schema/docs/views, and rebuilds the SQLite FTS5 table item_fts.
Existing folios can rebuild search with bin/console folio:fts:rebuild <provider/dataset> --query="search terms".
FolioArchiveService refreshes archive metadata before packaging.
FTS tables are derived data. Published archive files may drop item_fts, VACUUM, compress, ship, then rebuild FTS on the consuming side.
SQLite views and docs are also derived from persisted metadata, but they are intentionally lightweight and useful for standalone consumers.
Vector search is intentionally deferred. When added, start with a hybrid SQLite design: FTS5/BM25 for exact keyword strength, sqlite-vec for semantic retrieval, and Reciprocal Rank Fusion to merge ranks without normalizing incompatible score scales. Reference: https://ceaksan.com/en/hybrid-search-fts5-vector-rrf

Direct SQLite Examples

select * from schema_table where kind = 'dto';
select * from schema_property where table_id = ? order by position;
select local_id, label, dto_type, dto_data, extras from item limit 20;
select * from dto_document limit 20;
select id, type, audience, body from docs order by position;

TODO

Add fieldSet support to the api-grid spreadsheet view to avoid displaying every DTO field at once.
Rebuild views/docs on restore, not only FTS, if the archive was packaged without them.

Catalog consumers and published shared folios

survos_folio.read_only: true opens FolioService contexts using SQLite mode=ro, disables schema auto-migration/folio-row creation and avoids WAL mode changes. Publish a compatible, completed, checkpointed folio before sharing; the reader will not upgrade it. Explicit archive restore and inflate commands still write their selected local targets.

archive_api_prefix defaults to /folio and controls folio:pull listing/fallback download URLs independently of this app's route_prefix or a remote /f browse prefix. Download URLs returned by the catalog remain authoritative. local_passthrough: true skips already-present shared folios even with --force.

PeriodicalCoverageService::compute() preserves periodical block/article/ad distinctions and returns counts plus per-issue details. It scans folio rows: run it offline when recording catalog summaries, never as a directory-page cache-miss fallback. zm's publisher command persists the compact versioned summary in its registry; Ink consumes that recorded API data.

Catalog UI

The former dataset-registry collection controller and grid are removed. Applications render published catalog entries using FolioCatalogClient; the demo shows this without a dataset database. Reader routes for individual folios remain available.

survos / folio-bundle

Package info

Security

Fund package maintenance!

Statistics