README

Consume Harvest's catalog and selected folios over HTTP. Persist the catalog in your application's existing database, without installing dataset-bundle or connecting to Harvest's database.

Ownership and storage

survos/dataset-bundle owns the producer registry and pipeline in Harvest.
survos/harvest-bundle owns the consumer's catalog, selection and sync checkpoint.
survos/folio-bundle opens, reads, archives, restores and downloads individual folio files. Their contents remain in those files, not in the application DB.

The bundle extends AbstractSurvosBundle and uses kit's HasDoctrineEntities. The four entities in src/Entity auto-register with the default entity manager:

Entity	Table	Purpose
`Dataset`	`harvest_dataset`	Remote identity, source, provider and public metadata snapshot; `row_count`, `tags`, `content_type` and `locales` are also columns so the app can sort and facet in DQL
`Artifact`	`harvest_artifact`	Each artifact type/code, remote revision, URL and last locally acquired revision
`SetMembership`	`harvest_set_membership`	Recorded dataset membership and download policy per named set
`HarvestSyncCheckpoint`	`harvest_sync_checkpoint`	Last successfully applied feed cursor and dataset map

No connection or entity manager is created. The application owns its schema migrations. Producer pipeline fields are not modeled here; arbitrary artifact types are tracked, while file acquisition currently supports folios only.

Install and configure

Requires PHP 8.5+, Symfony 8.1+, Doctrine ORM, kit-bundle and folio-bundle. API Platform and dataset-bundle are not dependencies.

composer require survos/harvest-bundle

Flex registers Survos\HarvestBundle\SurvosHarvestBundle; without Flex, register it and its installed dependency bundles in config/bundles.php. Options are defined in SurvosHarvestBundle::configure(). Configure selections in the application's config/packages/survos_harvest.yaml:

survos_harvest:
    folio_sets:
        tours:
            label: Tours
            download: true
            criteria:
                provider: [curatescape]
                minRows: 1

Set download: false to record membership and metadata without acquiring files. When sets overlap, an artifact is downloaded if any matching set requests it. Supported criteria are provider, tags (any), tagsAll, contentType, minRows and explicit folios (dataset keys). Metadata criteria intersect; explicit keys additionally select those datasets. A missing explicit key fails loudly. Empty criteria select nothing. All remote datasets and artifact metadata are stored in the local catalog; sets determine membership and file acquisition.

The default server is https://harvest.survos.com. An absent or empty HARVEST_SERVER uses that default. Override it in .env.local for development:

HARVEST_SERVER=https://127.0.0.1:8011

Folios land under APP_DATA_DIR (folio-bundle's data_dir). Published folios are public, so no token is needed; HARVEST_READ_TOKEN is optional and reserved for private folios. A token that is sent must be valid. You can also configure server and read_token directly. The bundle configures folio-bundle's dataset API and server for downloads; remove conflicting legacy survos_folio.folio_server or dataset_api settings. local_passthrough: true uses an existing shared folio volume; it never shares a registry database. lock_factory defaults to lock.factory and must use a store shared by sync workers for this application. The lock key is derived from the application's database, so separate apps on one machine (each with LOCK_DSN=flock) never block each other's syncs.

Preview, migrate and sync

bin/console harvest:sync --dry
bin/console doctrine:migrations:diff
# Review the generated migration, then apply it:
bin/console doctrine:migrations:migrate
bin/console harvest:sync
bin/console harvest:sync:status

--dry reads Harvest and any existing local checkpoint, evaluates sets and prints the proposed selection. It writes no catalog, checkpoint, membership, lock or folio files and performs no downloads. With no Harvest tables, it previews an initial full sync without creating schema. --full forces complete reconciliation.

Use repeatable --folio=provider/dataset options to acquire only specific folios already selected by the application. Every selected variant of those exact dataset keys is eligible; unknown or unselected keys fail before acquisition or commit. The full catalog and set membership are still reconciled. Excluded folios are shown as outside scope, are not opened or downloaded, and receive no new acquisition receipt. Existing same-source receipts remain unchanged; changing servers still discards the old source's receipts. This option does not verify or repair earlier acquisitions. A later unscoped run can acquire pending folios. --limit counts only eligible new or changed downloads, and --dry previews both options without writes.

bin/console harvest:sync --dry --folio=cron-america/sn84030143 --folio=cron-america/sn90059960

A real sync locks, fetches the current catalog/change feed, acquires selected folios, then commits dataset metadata, artifacts, membership and checkpoint in one database transaction. A failed download or database write leaves the cursor unchanged. Downloads are outside the DB transaction and may already exist after a later failure; retries are idempotent. Removed remote records/membership are reconciled, but local folio files are never deleted automatically. Uncompressed artifacts remain tracked; acquisition checks for an existing local file until Harvest publishes an archive.

Inject Survos\HarvestBundle\Catalog\HarvestCatalog and call members('tours') to get FolioCatalogEntry objects from recorded membership. This read makes no HTTP call. Migrate and run an initial sync before switching application pages to the local catalog.

Optional local DuckDB preparation

Enable this in a consumer with native PDO DuckDB 2.x installed:

survos_harvest:
    duckdb:
        enabled: true
        fts: true

harvest:sync uses the existing archive download and restore path, then prepares <dataset>.duckdb alongside <dataset>.folio before recording acquisition. Both options default to false. FTS requires the DuckDB fts extension to be installed for the PHP runtime; preparation loads it but does not install it. The optional index covers the search projection's title and description.

Preparation belongs to folio-bundle, and generic FTS indexing to search-bundle. It copies the folio's logical tables, creates typed DTO tables and a search projection, then checkpoints and atomically publishes the completed file. It uses a per-target local lock and preserves the previous companion on failure. The archive and Doctrine ORM access remain unchanged.

Every acquired selection is checked, including unchanged revisions. A matching shared build ID, DTO schema signature, exporter/projection versions and FTS option reuse the file; missing or stale companions are rebuilt. A conversion error fails sync before its catalog checkpoint is committed, so the next run retries it. --dry, deferred downloads and sets with download: false never prepare files. Older archives without _folio_build must be rebuilt by the producer first.

This is synchronous. It neither uploads the companion nor changes local/remote search selection. A future dev action can call the same acquisition service.

Filtered item sources

Folio selection and item selection are separate. criteria continues to determine set membership and download continues to control acquisition. item_criteria filters the items streamed from those members; it changes neither their membership nor the underlying folio files.

survos_harvest:
    folio_sets:
        withImages:
            criteria:
                provider: [nara]
            download: false
            item_criteria:
                minImages: 1
        mistralHandwriting:
            criteria:
                provider: [nara]
            download: false
            item_criteria:
                tags: ['ai:handwriting:mistral']

minImages counts attached photo, scan, and document pages with non-empty URLs. Audio/video pages and thumbnail metadata do not count. A folio without a page table has zero images. The default is 0. tags matches any listed tag with exact, case-sensitive equality; its default empty list imposes no tag condition. Both predicates must pass when both are configured. Canonical item tags take precedence over legacy extras.source_tags.

Inject Survos\HarvestBundle\Set\SetItemSource into an application indexer or importer:

$report = new \Survos\FolioBundle\Service\FolioDocumentStreamReport();
foreach ($source->documents('mistralHandwriting', keep: ['title', 'tags'], report: $report) as $item) {
    // Feed your index or database. Use rowId as the original source identity.
    $indexer->upsert($item['rowId'], $item);
}
if ($report->failed !== []) {
    // Handle incomplete input before replacing an index or deleting absent records.
    throw new \RuntimeException(json_encode($report->failed, JSON_THROW_ON_ERROR));
}

The source reads recorded membership from the application's catalog and streams only local folio files using the existing FolioDocumentStream. It never syncs or downloads; missing/unreadable files are recorded in report.failed. Unknown set names throw. Source/locale artifacts do not duplicate items: locale: 'en' explicitly selects the local English variant, otherwise the source folio is used. The set's core defaults to obj. Filtering happens before keep projection and report counts include only yielded items. rowId, folioCode, coreCode, and localId retain provenance. Applications own batching, storage and index updates.

These filters are opt-in through SetItemSource; existing folio browsers and commands that read raw membership are not implicitly filtered.

Administration UI

Visit /admin/harvest/ as a user with ROLE_ADMIN. The local catalog, dataset details, artifact revisions and folio-set membership are read from the app DB. Preview sync and Sync now use the same service and lock as the console command. Both actions require a CSRF-protected POST. Sync runs within the web request; use the console for acquisitions that exceed your web request timeout.

Search Harvest API queries Harvest from the application server using its configured read token, without sending that token to the browser or requiring a Harvest login. Search filters the paginated remote catalog by text and provider; the catalog is cached for five minutes, with an explicit refresh. Results include dataset keys for adding to folio-set configuration. Searching does not change membership or download files. Help & folio sets explains selections, track-only sets, sync, setup and the application's current selection rules.

With Tabler installed, a Harvest menu is added to ADMIN_NAVBAR_MENU for admins. Other layouts can link to the survos_harvest_admin route, as Rutado's Settings page does. Templates extend base.html.twig; override base_layout if needed. Kit's routes_enabled and route_prefix options control route registration; set admin_navbar_menu: false to omit the menu entry.

Migration status

Rutado is the first consumer integration. Published 2.37.1 was validated there with local migrations, a dry run and a real sync. The administration UI described above is included starting with 2.37.2.

Existing dataset-bundle sync services and folio-bundle configuration remain available for unmigrated consumers. Do not enable the old survos_dataset.harvest_sync alongside this bundle: both expose harvest:sync. Remaining consumers can migrate independently; legacy dataset-bundle sync must be disabled first. The old Harvest attribute and metadata registry are deprecated scaffold classes, not the persistent catalog.

Validation and release

# In mono
vendor/bin/phpunit -c bu/harvest-bundle/phpunit.xml.dist
composer validate --strict bu/harvest-bundle/composer.json
# Standalone package checkout
composer install
composer test

Source lives in survos/mono, under bu/harvest-bundle, already registered in .github/workflows/split.yml. Coordinate releases with the release owner; never replace a published tag.

Scheduled sync and application projections (next patch)

The bundle registers Symfony's harvest schedule by default. It runs harvest:sync every two minutes (schedule_interval, or the HARVEST_SYNC_INTERVAL env var, e.g. "30 seconds"), retaining its schedule state and replaying only the last missed run. Set schedule_enabled: false to opt out. A set that names a folio Harvest does not list logs a warning; the rest of the sync proceeds. Run a persistent worker:

bin/console messenger:consume scheduler_harvest --time-limit=3600 --memory-limit=256M
bin/console debug:scheduler harvest

For Dokku, declare that command as a Procfile process and scale it to one worker; a schedule definition alone does not start a background process. Use a shared Symfony Lock store across web, scheduler and CLI instances.

survos_harvest.sites owns the host-to-set mappings, and each set can specify its presentation core (default obj). These are forwarded to folio-bundle's existing presentation services for compatibility. Their recorded membership is read from the application database through RecordedMembershipProvider, with no file-cache or HTTP fallback. catalog_ttl also belongs under survos_harvest.

Survos\HarvestBundle\Event\HarvestSyncedEvent is dispatched after membership is flushed, inside the transaction and before advancing the checkpoint. Listeners can read the new membership and update app-owned projections using the default entity manager; throwing rolls back both projections and the catalog/checkpoint. Listeners run on every real sync so configuration or local-file changes are applied without requiring a remote revision change. Dry runs do not dispatch the event. Keep irreversible external effects out of these listeners.

Application-owned selection

Apps whose database determines which folios to acquire can implement Survos\HarvestBundle\Sync\HarvestSelection and alias that interface to their service. plan() receives the complete reconciled checkpoint and returns a SyncPlan containing selected artifacts, download flags and set membership. Read application selection on every call, retain the complete checkpoint, and keep planning free of writes/downloads so harvest:sync --dry stays safe. The default implementation remains the configured folio_sets selector.

Remote translated DuckDB snapshots

With folio-bundle 2.45.16+, metadata-only sync records published DuckDB languages, immutable S3 addresses and revisions in the existing catalog tables. No schema change or local snapshot is required. Configure the consumer:

survos_harvest:
    duckdb:
        remote: true
        default_locale: en
        s3:
            endpoint: '%env(HARVEST_S3_ENDPOINT)%'
            access_key: '%env(HARVEST_S3_ACCESS_ID)%'
            secret_key: '%env(HARVEST_S3_ACCESS_SECRET)%'
            region: fsn1
    folio_sets:
        suisse:
            download: false
            criteria: { folios: [fpeu/suisse] }

Run harvest:sync --folio=fpeu/suisse (the scope avoids acquisitions requested by other sets), then open /folio/fpeu/suisse/duckdb/search?language=es or en. Apps with locale_prefix: true prepend their UI locale, for example /en/folio/.... All sets matching Suisse must have download: false. The native PDO DuckDB 2.x and httpfs extensions are required. Credentials stay in application environment configuration and are never obtained from the public catalog.

Remote mode selects exclusively from the consumer's synced metadata. Missing snapshots or languages return 404, rather than silently using a local file or a different language. Language selection persists per dataset in the browser session for live searches; tabs share that selection. Snapshot replacements and withdrawals follow normal sync. HarvestCatalog::remoteSnapshots() exposes the typed entries. Existing SQLite readers and local acquisition remain available; remote mode wires the optional DuckDB routes, it does not translate an application's custom SQLite SQL.

survos / harvest-bundle

Package info

Security

Fund package maintenance!

Statistics