README

A pinned, reproducible Drupal site with known ground-truth state, used as the target AI agents are evaluated against. You point an agent at a real Drupal, let it build, and grade what it actually built: the saved config, the executed view, the persisted Canvas tree. Not what the model says it did.

Project page: drupal.org/project/ai_proving_ground.

Why this exists

Most AI-agent evals for Drupal run in an empty folder. They grade the text the model produces, so a plausible answer scores well even when the build it describes would not work. The teams building Drupal's eval tooling (ai_eval, ai_bench, the Inspect AI runner, the behavioral skill evals) all hit the same missing piece: a shared, resettable site with known contents that graders can check against. This project is that piece.

Three properties make a site usable as an eval target, and they are the whole design:

Known state. Every content item and config object the graders rely on is pinned in a fixture: fixed UUIDs, recorded fields, declared counts. A doctor command verifies the live site still matches, and says exactly what drifted when it does not.
Cheap reset. Runs mutate the site; the next run needs the same starting point. Provisioning captures a baseline (a database dump plus a git ref) keyed by a content hash of the exact fixture set, and reset restores it in seconds. Same set, same baseline, every time.
One grader, both directions. The droost_state grader from ai_eval_droost (the Droost-backed grader, a submodule of ai_eval since 1.0.0-beta2) reads live state through Droost's read and validate tools. The ai_eval runtime calls it in-process (inside-in). An external harness such as Inspect AI calls the same grader over drush through ddev ai-eval-grade and gets an every_eval_ever 0.2.2 envelope back (outside-in). The scores are comparable because the grading code is identical.

How it fits together

The bed is not one fixed site. It is a base recipe (Canvas plus the eval stack) and a library of fixture recipes. Each fixture seeds one coherent content and config set and declares what it owns: a set of config name prefixes and a content-UUID namespace. A target names the fixtures it needs, pinned as name@version; the bed composes them and refuses to compose fixtures whose ownership claims overlap.

recipe.yml            # BASE recipe: Canvas + eval stack, no target content
fixtures/
  fixture_kb_docs/    # a fixture = recipe + config + content + manifest fragment
    recipe.yml
    config/           # pinned config objects (exported, not hand-written)
    content/          # pinned content, fixed UUIDs (core default-content format)
    manifest/fragment.yml   # what this fixture OWNS + machine-written facts
  fixture_canvas_home/
manifest/             # example of the unioned manifest the doctor reads
ddev/
  commands/           # the five ddev commands (see below)
  agents/             # run adapters; the shipped one drives nothing itself
scripts/              # hash, fixture-export, manifest union, doctor, task, grade
skills/               # operator skills: extend the bed and run evals
docs/                 # design amendments and live verification evidence

Two fixtures ship today. kb_docs is a small knowledge base: the kb_doc content type, four fields, and 12 published nodes whose sections and titles the graded tasks depend on. canvas_home is the site's front page as a Canvas page entity plus its two page regions. Together they back the first graded dataset (a Views build task and a Canvas build task, in ai_eval_droost's droost_build_tasks dataset).

Getting started

The outer bootstrap is delegated to one_line_installer: it stands up DDEV and Drupal CMS (which includes Canvas), unattended. These are the steps on the project page. The first end-to-end run is recorded in docs/disposable-bed-proof.md.

# 1. A fresh Drupal CMS site with droost, fully unattended.
#    Keep --agents none. Without it, the installer adds Claude Code and
#    mounts your host ~/.claude, login included, into the web container,
#    where the agent you test can read it.
bash <(curl -fsSL https://aibp.drupalstarforge.ai/install.sh) \
  --yolo --name my-bed --extras droost --agents none
cd my-bed

# 2. The eval stack (AI Eval includes ai_eval_droost) and this recipe,
#    which Composer installs to recipes/ai_proving_ground.
ddev composer require drupal/ai_eval drupal/ai_proving_ground

# 3. The DDEV commands and agent adapters.
mkdir -p .ddev/commands/web .ddev/commands/host .ddev/agents
cp recipes/ai_proving_ground/ddev/commands/web/ai-eval-* .ddev/commands/web/
cp recipes/ai_proving_ground/ddev/commands/host/ai-eval-* .ddev/commands/host/
cp recipes/ai_proving_ground/ddev/agents/* .ddev/agents/
chmod +x .ddev/commands/host/ai-eval-* .ddev/agents/*

# 4. Baselines need a git tree. The installer does not create one:
# Drupal CMS ignores /recipes; baselines need the whole directory tracked.
sed -i '/^\/recipes$/d' .gitignore
git init -b main && git add -A && git commit -m "bed initial state"

# 5. Provision: base recipe + fixtures + baseline capture.
ddev ai-eval-provision kb_docs canvas_home

# 6. Verify, then use.
ddev ai-eval-doctor kb_docs canvas_home
ddev ai-eval-grade dv01_view_build

From there the loop is: reset, let the agent act, grade, reset. The middle step has a paved path, ddev ai-eval-run <task>, described under Running an agent below; nothing stops an external harness from driving the bed itself.

Commands

Commands take the fixture set as arguments (default kb_docs canvas_home).

Command	Job
`ddev ai-eval-provision [fixture...]`	Apply the base recipe and the named fixtures, then capture baseline B for that set: a DB dump plus a git tag, keyed by the content hash of base version + sorted `fixture@version` list. No-op when that hash already has both halves of its pair.
`ddev ai-eval-reset [fixture...]`	Restore baseline B for the set: Composer code when it differs (`composer install` from the recorded files), then `settings*.php` and edited packages (`composer reinstall`), then the scoped git tree (never `vendor/`), then the DB dump, then what an attempt added to the scratch paths, then `drush cr`. Derives the same hash in plain PHP, so it works when Drupal cannot boot; never falls back to a naive key.
`ddev ai-eval-run <task-id> [--agent=<name>] [--dataset=<ref>] [--model=<id>]`	Resolve a task to its prompt, hand it to an agent adapter, and record the attempt in a run manifest. Never grades. See Running an agent below.
`ddev ai-eval-claude token \\| check \\| shell`	Make and test the scoped token for the `claude` adapter, or open an isolated Claude Code session in the web container. See The Claude Code adapter below.
`ddev ai-eval-grade <task-id> [dataset] [model-id] [--run=<run-id>]`	Grade one task against live state with the `droost_state` grader and print an every_eval_ever 0.2.2 envelope on stdout, then store it so the verdict appears in ai_eval's results UI (`AI_PG_NO_IMPORT=1` prints without storing; see Where verdicts go below). Exit 0 means a verdict was emitted, pass and fail alike; exit 3 means the question has no droost spec so nothing was graded (stderr carries `APG_GRADE_SKIP`); other non-zero means the grade could not run. Read the exit-code caveat below before relying on 3. The optional model-id stamps `model_info` (outside-in, only the driving harness knows which model acted).
`ddev ai-eval-diff [fixture...] [--run=<run-id>]`	Print one JSON report of what changed since baseline B: packages, Composer files, package file edits, settings files, custom and tracked files, modules, themes and config. Run it after the attempt and before reset. It changes nothing. `--run` also saves the report as `diff.json` in the run directory. See What an attempt changed below.
`ddev ai-eval-doctor [fixture...]`	Assert live state matches the union of the set's manifest fragments (installed modules, entity counts, every declared UUID resolving with the declared label) and that no two fixtures claim the same config prefix or UUID namespace. JSON verdict, exit 0 when healthy.

Bed git requirements

Reset only restores files git tracks in the baseline ref under config/sync, web/modules/custom, and recipes. A stock Drupal project must remove or override the /recipes ignore rule, then add and commit the recipe files before provisioning. Absent or empty paths are allowed. A populated path with no files tracked in the baseline is refused with APG_PROVISION_UNTRACKED or APG_RESET_UNTRACKED; the diagnostic identifies the path and matching ignore rule.

web/modules/custom and web/themes/custom are scratch paths when git does not track them at provision, as on a stock Drupal CMS bed. Provision records the files they hold, and reset removes what an attempt added there after it restores the database, instead of refusing with APG_RESET_UNTRACKED. Files that were there at provision are kept as they are, not restored. Track the folder in git to have reset restore its content.

Set AI_PG_ALLOW_UNTRACKED=1 inside the web container to accept DB-only restoration for untracked paths. This prints a warning and does not disable grader checks. Provision records grader.<set>.sha256 and grader.<set>.files beside the DB dump. The recipe and the Drupal-resolved ai_eval (including ai_eval_droost) and droost module trees are fingerprinted, excluding .git and .ai-proving-ground state/run output. Contrib grader code is checked, not restored. Reset fails with APG_RESET_GRADER_DRIFT and changed paths if the restored bed still differs; doctor reports a failing grader_tree check. Do not grade that bed.

Older baselines without a fingerprint remain usable: reset warns with APG_RESET_GRADER_UNRECORDED, and doctor reports a non-failing "not recorded, re-provision to record" note. To record one, remove the matching dump and baseline tag as described by provision, then provision again from the accepted clean tree.

Code upgrades

The fixture-set hash stays unchanged across code upgrades. Each new baseline records versions.<set>.json beside its dump: Drupal core and PHP versions, modules from the base and selected fixture recipes, and ai_eval, ai_eval_droost, and droost. Module code identity uses the containing Composer package name, version, and source reference when available, then the module's info.yml version, then unknown. Core module code versions are covered by the core entry. Installed database schema versions are recorded for all these modules for diagnosis, but are excluded from code comparisons.

On an existing baseline, provision refuses code differences with APG_PROVISION_STALE_BASELINE and lists recorded and live versions. AI_PG_ALLOW_STALE_BASELINE=1 warns and preserves the existing no-op; it does not refresh the dump or manifest. Older baselines without a manifest warn with APG_PROVISION_VERSIONS_UNRECORDED. Doctor's code_versions check fails on code differences; missing manifests are a non-failing "not recorded, re-provision to record" detail.

Provision also records the Composer state for the set: copies of composer.json and composer.lock and the installed package list from vendor/composer/installed.json, as composer.<set>.json, composer.<set>.lock and composer.<set>.state.json. On an existing baseline, a Composer difference is refused with APG_PROVISION_STALE_BASELINE like a version difference, and AI_PG_ALLOW_STALE_BASELINE=1 keeps the existing baseline. Doctor reports it in the composer_state check.

Provision also records site.<set>.json (the content of web/sites/*/settings*.php except DDEV's own settings.ddev.php and the scratch path listings) and site.<set>.packages.json.gz (size, mtime and SHA-1 of every file in every installed package). Reset puts back the settings files, removes added ones, and runs composer reinstall on each package whose files changed; it fails with APG_RESET_PACKAGE_DRIFT if they still differ. A package installed from source (a VCS checkout) is never reinstalled, because its changes can be your own work: reset prints APG_RESET_PACKAGE_SOURCE_EDITED and goes on, and the grader check still fails for AI Eval and Droost. Doctor reports all of this in the site_files check. Older baselines without these files warn and go on; re-provision to record them.

Reset restores Composer code first, before the database. When the files or the packages differ (APG_RESET_COMPOSER_CHANGED lists each difference), it copies back the recorded files and runs composer install, which also removes packages the attempt added. If the packages still differ afterwards, reset stops with APG_RESET_COMPOSER_DRIFT. A baseline from before this check warns that Composer code was not restored; re-provision to record it.

After restoring the database and rebuilding caches, reset warns about code differences with APG_RESET_CODE_DRIFT. It checks pending database updates and fails with APG_RESET_PENDING_UPDATES, listing modules and updates, if the restored database is behind the code. It never runs updates automatically. Do not grade a bed after a failed reset.

After upgrading code, bring the disposable bed's database up to date explicitly with ddev drush updb -y, and verify the intended fixture state. Remove the matching dump and baseline tag named in provision's recapture instruction, then run ddev ai-eval-provision with the same fixtures to capture a new baseline and manifest. This reapplies recipes; if the existing state cannot safely be reapplied, rebuild the disposable bed first. Run ddev ai-eval-doctor afterward. Keep a backup of the old baseline pair if you need to return to the old code.

Where verdicts go

ddev ai-eval-grade prints its envelope on stdout and also stores it, so an outside-in verdict shows up at /admin/config/ai/ai-eval/results beside the runs ai_eval drove itself. That is the comparison the grading seam exists to make: same grader, both directions, one screen. The grade already runs inside Drupal, so it hands the envelope straight to ai_eval's importer with no trip through a file.

Storing can never change the outcome. It happens after stdout is finished, it reports only on stderr, and it swallows its own failures: a storage problem must not masquerade as a grading failure, nor turn an emitted verdict into a non-verdict. If the envelope cannot be stored, the verdict on stdout still stands and stderr reports APG_GRADE_NOT_STORED <reason>. This includes importer exceptions, rejections, and conflicts. Exit 0 still means a verdict was emitted. An intentional skip or absent importer does not emit this token.

ddev ai-eval-grade dv01_view_build --run="$RUN_ID"  # grade a recorded attempt
ddev ai-eval-grade dv01_view_build              # print and store
AI_PG_NO_IMPORT=1 ddev ai-eval-grade dv01_view_build   # print only

Set AI_PG_NO_IMPORT=1 when an external harness owns persistence, or when you are grading repeatedly and do not want the rows.

Two things worth knowing before you rely on it:

Storing needs a new enough ai_eval. The importer arrived in ai_eval issue #3594739. On an older ai_eval the grade prints as it always did and says once on stderr that it could not store. The base recipe does not pin a version that guarantees it, so this is a normal state rather than an error.
Run identity controls deduplication. With --run, the evaluation ID ends in the run ID. An identical re-grade is already stored. A changed measurement for that ID is an import conflict. Without --run, the ID ends in a timestamp and four random bytes encoded as hex.

--run=<run-id> can appear anywhere in the arguments. Grade reads run.json from ${AI_PG_BASELINE_DIR:-<approot>/.ai-proving-ground}/runs/<run-id>/. It uses the recorded model when the model argument is empty. A conflicting model, a different task, an invalid run ID, or missing or unreadable run metadata emits APG_GRADE_RUN_INVALID <reason> on stderr. No envelope is emitted and the wrapper exits 1.

The envelope records apg_run_id in source_metadata.additional_details. Grade also saves it as grade.json in that run directory using a temporary file and rename. Re-grading replaces that file. A failed save emits APG_GRADE_NOT_SAVED <reason> on stderr but does not fail the grade. The run directory, including run.json and grade.json, survives ai-eval-reset. Reset restores the database and only config/sync, web/modules/custom, and recipes.

Check hashes in the envelope

source_metadata.additional_details records:

dataset_path: the FileSource-selected file, relative to the Drupal app root when inside it, otherwise absolute.
dataset_sha256: SHA-256 of that file's bytes.
task_sha256: SHA-256 of the matching questions[] entry parsed from those same bytes. Recursively sort associative-array keys, preserve list order, then JSON-encode with JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRESERVE_ZERO_FRACTION | JSON_THROW_ON_ERROR.

eval_library.additional_details records:

grader_class_file: the resolved droost_state class file, relative to the app root when inside it, otherwise absolute.
grader_class_sha256: SHA-256 of that class file.
grader_tree_sha256: the grade-time fingerprint from the shared grader-tree helper, covering the recipe and installed ai_eval/droost trees.
hash_algorithm: sha256.

Verify file hashes with sha256sum <dataset file> and sha256sum <class file>. Recompute the canonical task hash to compare one check across dataset edits. Compare grader_tree_sha256 with the bed's .ai-proving-ground/grader.<set>.sha256 (or the escrow's copy). Provision must have recorded the same code; installing new grader code changes that fingerprint. If dataset resolution, parsing, or the loaded task's ID/droost cross-check fails, all three dataset keys are omitted and dataset_provenance says unresolved: <reason>. A tree failure omits its hash and sets grader_tree; a class-file failure sets grader_class. Failures warn on stderr with APG_GRADE_DATASET_UNRESOLVED or APG_GRADE_GRADER_UNRESOLVED and do not change the verdict.

These hashes do not authenticate the envelope: it is agent-writable after the fact, so the harness must keep the stdout it captured (see the escrow section). grader_tree_sha256 is computed by code inside the bed and is only as trustworthy as an ai-eval-verify run before grading.

Running an agent

ddev ai-eval-run <task> closes the gap between reset and grade. It resolves the task through the same dataset loader ai-eval-grade uses, so run and grade can never disagree about which task is which, writes a run directory under .ai-proving-ground/runs/, and hands the prompt to an adapter. It never grades: a failed attempt and a failed grade are different facts.

Two properties are non-negotiable for an eval bed, and the command enforces both rather than asking you to remember them.

Vanilla by default. The default adapter, manual, drives nothing. It prints the prompt and waits for you to run whatever agent you are actually evaluating. A bed that shipped a tuned agent, skill set or rule set would be grading its own configuration; tuning is what this bed measures. Selecting a real runtime is always explicit (--agent, or AI_PG_AGENT). The one runtime adapter that ships, claude, adds no skills, settings or rules of its own (see The Claude Code adapter below). With no terminal attached, manual records no attempt and exits non-zero rather than claim one happened.

Bed-scoped credentials. A bed runs untrusted attempts by definition, so an agent running against it must not hold the host's credentials. Before running an adapter that needs credentials, the command inspects the bed's own DDEV config for host-wide credential sharing and refuses with APG_RUN_CRED_REFUSED when it finds any. The preflight detects volume sources outside the project (home paths, variable expansions unless known to resolve inside the project, and absolute host paths), host Docker socket mounts, and credential-shaped environment names in config*.yaml, .ddev/.env, and .ddev/.env.* (excluding .env.example). It reports names and source paths, never secret values. AI_PG_ALLOW_HOST_CREDS=1 remains the deliberate override. This is not hypothetical: a bed built by one_line_installer with the ddev-assistant-claude add-on bind-mounts the whole host ~/.claude, credentials included, into the web container.

An adapter is any executable in .ddev/agents/ (or the recipe's ddev/agents/). It declares its credential needs in a header line the same way a ddev command declares its description:

#!/usr/bin/env bash
## Description: what this adapter drives
## Credentials: none | bed-scoped | host-shared

A missing or unrecognised declaration reads as unknown and is treated as needing credentials. Fail closed: a bed that guesses in the permissive direction is the exact failure this project exists to catch. The adapter receives the run request by environment and file, never on stdin, so an interactive agent CLI keeps the terminal:

Variable	Meaning
`APG_RUN_DIR`	the run directory; write anything worth keeping here
`APG_PROMPT_FILE`	the task prompt, verbatim
`APG_REQUEST_FILE`	the full run request as JSON
`APG_TASK_ID`, `APG_DATASET`, `APG_MODEL`	what is being attempted
`APG_APPROOT`, `APG_DDEV_PROJECT`	the bed to act on

Exit 0 from an adapter means an attempt completed. Anything else is recorded as no attempt.

.ai-proving-ground/runs/<run-id>/run.json is the authoritative record of a run: task, dataset, adapter, model, timestamps, adapter exit code, whether an attempt happened, the prompt's sha256, and what the credential preflight found. It sits beside the baselines, which ai-eval-reset does not restore, so the record of an attempt survives the reset that follows it. Read it rather than parsing terminal output, and cite it from an envelope when a run needs provenance.

The Claude Code adapter

claude runs Claude Code headless in the web container, one attempt per run:

ddev ai-eval-run dv01_view_build --agent=claude --model=claude-sonnet-5-5

It needs a scoped token, not your login. ddev ai-eval-claude makes it and gives you the same isolated Claude Code by hand. You need no Claude Code on your computer: all three run in the web container.

Command	What it does
`ddev ai-eval-claude token`	Runs `claude setup-token`. Open the link and log in. The browser shows a code: paste it into the terminal. Claude Code then prints the token, which starts with `sk-ant-`. Paste the token at the `Token:` prompt (hidden). The command saves it in `~/.config/ai-proving-ground/claude-token` with mode 600 and tests it. It refuses text that is not a token (`APG_CLAUDE_NOT_A_TOKEN`). `--paste` skips `setup-token` when you already have a token.
`ddev ai-eval-claude check`	Tests the saved token with one short call.
`ddev ai-eval-claude shell [claude args]`	Claude Code, interactive, with the isolation of an attempt (see below). Use it to drive an attempt by hand with the `manual` adapter, or to see what the agent sees.

APG_CLAUDE_TOKEN_FILE sets another path. CLAUDE_CODE_OAUTH_TOKEN in the environment of ddev ai-eval-run also works. The adapter refuses a token file inside the project or one that other users can read. It never reads or copies ~/.claude.

What the agent gets:

the project files of the bed, such as CLAUDE.md, .claude/skills and .claude/settings.json. They are part of what you test. A bed made with the installer has AGENTS.md, CLAUDE.md and 32 skills. To test Claude Code with no project files, remove them and commit before you provision. The adapter refuses to start when they differ from the last commit;
a new, empty config folder in the container /tmp for each attempt, so no user settings, skills, hooks, plugins, memory or MCP servers;
the tools in APG_CLAUDE_TOOLS (default Bash,Read,Write,Edit,Glob,Grep, so no web search or web fetch);
the token, on stdin only. It is never an argument, a .ddev setting or a file in the bed.

The first run installs Claude Code with npm in ~/.apg-claude in the web container, outside the project. APG_CLAUDE_VERSION pins a version. ddev restart removes the install, and the next run installs it again. The agent runs as the same user, so it can change that install. Restart the bed when you need a clean one.

The run directory gets transcript.jsonl (stream-json), stderr.txt, install.txt and claude.json (Claude Code version, model, turns, cost and status). The adapter replaces the token value with [REDACTED] in the transcript and in stderr. After the attempt it looks for the token value in the run directory and in each file that git shows as changed or new. When it finds it, it names the files and exits with APG_CLAUDE_TOKEN_LEAKED. Revoke that token. The scan does not look in the database or in git-ignored files; reset restores those.

Reset does not restore .claude, .agents, CLAUDE.md, AGENTS.md or CLAUDE.local.md, and Claude Code loads them. So the adapter refuses to start, with APG_CLAUDE_PROJECT_CHANGED, when they differ from the last commit of the bed. An earlier attempt cannot change what the next attempt gets. To test a change to them, commit it.

The adapter also refuses to start when the web container can use SSH keys from the host agent (ddev auth ssh), with APG_CLAUDE_SSH_KEYS. The run preflight still refuses a bed that shares host credentials before the adapter starts. APG_CLAUDE_TIMEOUT (default 3600 seconds) stops a long attempt.

Exit 0 means Claude Code reached a result that is not an error. The other outcomes have stable stderr tokens: APG_CLAUDE_NO_TOKEN, APG_CLAUDE_TOKEN_UNSAFE, APG_CLAUDE_SSH_KEYS, APG_CLAUDE_INSTALL_FAILED, APG_CLAUDE_AUTH_FAILED, APG_CLAUDE_TIMEOUT, APG_CLAUDE_FAILED, APG_CLAUDE_TOKEN_LEAKED, APG_CLAUDE_PROJECT_CHANGED and APG_CLAUDE_NOT_A_TOKEN. Run bash tests/claude-adapter.test.sh for the checks. It needs no Docker and no real Claude Code.

Versioning rule: bump a fixture's version: in its fragment whenever its exported content changes, and the base version under extra: ai_proving_ground: version in recipe.yml whenever the base changes. The hash does the rest: a bumped version is a new baseline, an unchanged set reuses the cached one.

Fixtures are exported, not hand-written

scripts/fixture-export.php runs on a source site and captures what the fragment declares: the owned config objects (exact names must exist, trailing-dot prefixes expand) and the content matched by the fragment's export.content selectors, in core's DefaultContent format with fixed UUIDs. It then regenerates the fragment's machine-written facts (counts, per-node records). Three keys stay hand-maintained: version, owns, and export. Ground truth is keyed on UUID, never on nid: node IDs are serials assigned at import time and they do not survive re-provisioning. The first disposable-bed run proved this the hard way; see finding 4 in docs/disposable-bed-proof.md.

To add a fixture or a graded task, read skills/extending-the-proving-ground/SKILL.md. It is written as an agent skill, so both people and coding agents can follow it.

Your own tests

Keep your fixtures in your project's recipes/fixture_<name>/, tracked in that project's git so they survive APG Composer updates and reset restores them. Alternatively, set AI_PG_FIXTURE_PATH to colon-separated directories that contain fixture_<name>/ folders. Relative entries are relative to the app root (DDEV_APPROOT, normally /var/www/html), not the Drupal docroot. APG commands run inside the web container, so set it there: ddev config --web-environment-add=AI_PG_FIXTURE_PATH=my-fixtures and ddev restart. Most projects do not need it: recipes/fixture_<name>/ works without any setting.

Lookup searches those entries, then the project's recipes/, then APG's shipped fixtures/. A hit requires recipe.yml. Two distinct directories with the same fixture name fail with APG_FIXTURE_AMBIGUOUS; use a new name for a variant. Symlinks to the same real directory count as one copy.

Export with scripts/fixture-export.php as described in the fixture guide. For a new name it creates an empty scaffold in the first AI_PG_FIXTURE_PATH entry, or in the project's recipes/ by default. Edit its recipe install list, ownership and export selectors, then re-export from the live source site. Existing fixtures export back to their resolved directory.

Put your tasks in a dataset YAML in your own enabled module's datasets/ folder (AI Eval 1.0.0-beta4 or later), or configure ai_eval.settings: dataset_path. Track the dataset in your project's git. Pick each test's fixture set with command arguments, AI_PG_FIXTURES, or an AI Eval target's pins via drush ai-eval:target-fixtures <target> --names. Custom search roots outside reset's tracked paths are not restored by reset.

Target binding

AI Eval 1.0.0-beta2 and later ship EvalTarget fixtures pins such as kb_docs@2 and drush ai-eval:target-fixtures <target>. The command prints name@version pins; add --names to pass the short names to APG:

ddev ai-eval-provision $(ddev drush ai-eval:target-fixtures my_target --names)

The target records the required versions; --names only selects the set, so compare the pins with the fixtures' manifest versions when updating them.

Test your own module

This section joins up what a module maintainer needs from the sections above.

1. Put the module on the bed before you provision

Provision records the Composer state and reset puts it back, so a module added after provision is removed by the next reset. Add it first:

Your code	Do this	What reset does with it
Released on drupal.org	`ddev composer require drupal/<module>`	Restores the recorded version
In progress, files only	Copy it to an untracked `web/modules/custom/<module>`	Keeps your edits to files that were there at provision; removes files added later. It also keeps an agent's edits to those files.
In progress, tracked in git	Commit it under `web/modules/custom/<module>`	Restores the baseline commit, so a code change needs a new provision

A Composer path repository is not tested yet.

2. Switch it on with a fixture

Make a fixture recipe in recipes/fixture_<name>/ whose install list enables your module and seeds the config and content your tasks need. Export it from a source site, do not write it by hand; see "Fixtures are exported, not hand-written" and skills/extending-the-proving-ground/SKILL.md.

3. Provision

ddev ai-eval-provision <fixtures>, then ddev ai-eval-doctor. Provision compares the versions of modules from the base and the selected fixture recipes, so a new release of your module is refused with APG_PROVISION_STALE_BASELINE until you provision again (see "Code upgrades").

4. Tasks and grading

Put tasks in a dataset YAML in your module's datasets/ folder (AI Eval 1.0.0-beta4 or later). ddev ai-eval-grade uses the droost_state grader, which reads Views (droost_views_get, droost_views_execute) and Canvas trees (droost_canvas_tree_get) only. A task with no droost spec exits 3 (APG_GRADE_SKIP). Grade other work with your own harness, for example an Eval Commons command check that runs in the DDEV container.

5. API keys and other secrets

There is no supported way yet. Know what captures what:

The database dump holds everything in the database at provision, including config and key entities. Escrow copies it.
site.<set>.json holds the content of settings*.php (not settings.ddev.php).
fixture-export.php writes the config objects the fixture owns.
Reset restores the database and settings*.php, so a key added after provision is gone after the next reset.
ddev ai-eval-run refuses credential-shaped names in .ddev/.env with APG_RUN_CRED_REFUSED.

Loop time

On a Drupal CMS bed with kb_docs and canvas_home, reset took 27 to 47 seconds (median 36) over 180 runs on 1.0.0-alpha2. A new provision takes longer.

Status

Alpha. The loop runs on new Drupal CMS sites with Canvas 1.12, with AI Eval 1.0.0-beta6 and 1.1.x. Grade envelopes validate against the reference every_eval_ever models. Baseline escrow and tamper-evident grading shipped in #3611944 and #3611940 (the evaluation that scoped them is docs/milestone-8-evaluation.md). Not yet done: the bug/fix rotation fixture, grading beyond Views and Canvas, and a supported way to give a module under test its API keys.

Verification evidence lives in docs/: milestone-3.md (baseline cache), milestone-6.md (grading seam), disposable-bed-proof.md (the full loop), milestone-8-evaluation.md (isolation options compared), and experiments/2026-08-13-view-discrimination/ (the worked A/B proof that a config-valid build can still be functionally wrong and only execution catches it, with envelopes, provenance hashes, and a copy-paste reproduce script that works on released versions, ai_eval 1.0.0-beta3 or later).

Implementation notes for contributors

DDEV collapses custom-command exit codes. ddev <command> returns 1 for every non-zero exit a custom command makes, host and web alike; the real code survives only as text on stderr (error=exit status 3). Verified on DDEV v1.25.1. ddev exec is faithful by contrast. So the documented codes above are what the scripts return, not what a caller of ddev ... observes: every distinguishable outcome also emits a stable stderr token (APG_GRADE_SKIP, APG_RUN_CRED_REFUSED, and so on), and a harness that needs a machine-readable result should read run.json or the grade envelope rather than an exit code.
All scripts run through `drush php:script --script-path=/scripts.php -- `; arguments arrive in `$extra`. Success paths fall through without `exit(0)`, because drush reports any explicit `exit()` as abnormal termination, and it collapses every explicit `exit(N)` to code 1. A script that needs to signal more than pass/fail emits a stable stderr token instead; the grade wrapper maps `APG_GRADE_SKIP` back to exit 3 outside the drush layer.
Recipe validation checks the install list against Drupal's cached extension list, so provision rebuilds caches before applying recipes.
On eCryptfs hosts, the DDEV bind mount can serve stale views of in-place file rewrites across the host/container boundary. Scripts here write new-file-then-rename. If an edited file looks stale on the other side, force a fresh inode: cp file .tmp && mv -f .tmp file.

License

GPL-2.0-or-later.

What reset restores

State	Reset behavior
Database	Restored from the baseline dump
Tracked paths	Restored from the baseline git ref
Public files	Restored from `files.<set>.tar.gz` to the recorded app-relative public directory
Private files	Not restored; reset prints `APG_RESET_PRIVATE_FILES_NOT_RESTORED <path>` when configured
Composer code (`composer.json`, `composer.lock`, `vendor/`, core, contrib)	Restored when it differs: reset copies back the recorded `composer.json` and `composer.lock`, runs `composer install`, and fails with `APG_RESET_COMPOSER_DRIFT` if the packages still differ
Edits inside package files	Restored: reset compares the files of every installed package with `site.<set>.packages.json.gz` and runs `composer reinstall` on each edited package; `APG_RESET_PACKAGE_DRIFT` if they still differ
`web/sites//settings.php`	Restored from `site.<set>.json`; settings files added since provision are removed
Scratch paths (`web/modules/custom`, `web/themes/custom` when git does not track them)	What an attempt added is removed after the database restore; files present at provision are kept as they are

Provision resolves public:// through Drupal and records the archive, a .json path/checksum sidecar, and a .list path listing beside the database dump. Top-level css/, js/, php/, and styles/ are regenerable derivatives: they are excluded from capture and doctor, and left untouched during reset. Doctor's public_files check compares paths, not file contents. Symlinks are archived as links and never followed during deletion. Stop concurrent writers before capturing or resetting a baseline.

Older baselines without a snapshot print APG_RESET_FILES_UNRECORDED and skip file restore; doctor reports a non-failing note. Provision's existing-baseline no-op does not backfill a snapshot. Recapture the baseline to enable file restore. Unsafe destinations or changed archives are refused before deleting public files.

What an attempt changed

ddev ai-eval-diff compares the site with baseline B and prints one JSON report. Use it after the attempt and before reset. A grader or a harness reads the report; it does not need its own before and after snapshots.

ddev ai-eval-diff kb_docs canvas_home --run=<run-id>

Report key	What it lists	Compared with
`packages`	packages `added`, `removed`, and `changed` (version or source reference)	`composer.<set>.state.json`
`composer_files`	`composer.json` or `composer.lock` when changed	`composer.<set>.state.json`
`package_files`	edited files of each installed package (`+` added, `~` changed, `-` removed)	`site.<set>.packages.json.gz`
`settings`	`web/sites//settings.php` added, changed, removed	`site.<set>.json`
`files`	files added, changed, removed in `config/sync`, `recipes`, `web/modules/custom` and `web/themes/custom`	the baseline git tag, and for scratch paths `diff.<set>.json`
`modules`, `themes`	`enabled`/`installed` and `uninstalled`	`diff.<set>.json`
`config`	config objects added, changed, removed, in every collection	`diff.<set>.json`

changed is true when any of these lists a change. An untouched bed reports false with every list empty. Exit 0 means a report was printed, with or without changes.

Provision writes diff.<set>.json: the enabled modules and themes, a sha1 of each config object, and a sha1 of each file in the scratch paths. A baseline from before this command has no such record. Then the report prints APG_DIFF_UNRECORDED drupal on stderr and modules, themes and config are null. The next ddev ai-eval-reset writes the record, because the site is at baseline B at the end of reset. When Drupal does not boot, the report still covers the files, and errors holds drupal: ... (also APG_DIFF_ERROR on stderr).

The command only reads the site. Its code is in the recipe, so an attempt can change it, the same as the grader. On an unattended bed, run it after ai-eval-verify and keep your own copy of stdout. Run bash tests/site-diff.test.sh for the checks. It needs no Drupal.

Agents that run the evals

The operator agent drives provision, reset, run, verify, and grade from outside the tested project. The agent under test works inside that project.

Install skills/running-proving-ground-evals/ for running and grading attempts. Install skills/extending-the-proving-ground/ for adding fixtures and tasks. Install both only on the operator side, in the skills folder of the operator's own workspace outside the tested project. Do not use ~/.claude/skills/ when the agent under test is Claude Code on the same host account: user-level skills load in every project. Never install them in the tested project's .claude/skills/ or .agents/skills/, or give them to the agent under test. They describe the checks and where expected answers live.

Unattended runs: escrow and verify

Provision a trusted bed, then save its baseline on the host:

ddev ai-eval-provision kb_docs canvas_home
ddev ai-eval-escrow kb_docs canvas_home

The host commands require Bash 4+, Git, jq, and GNU coreutils/findutils. Escrow writes to ${AI_PG_ESCROW_DIR:-$HOME/.ai-proving-ground-escrow}/<project>/<set-key>/. It captures the dump, available sidecars (missing ones produce APG_ESCROW_SKIPPED), a baseline-tag bundle, metadata, and a verifier copy. The host verifier captures its own judging-tree hashes, including .ddev/commands/ and .ddev/agents/. Other .ddev/ files are not fingerprinted. An existing set requires explicit --force to replace it. Only do this after reinstalling or re-provisioning trusted code, never to accept an attempt's drift.

Escrow setup calls ddev exec drush php:script ... hash.php with the selected fixtures; provision and reset run the same script with plain PHP. This uses the actual YAML parser instead of parsing human-readable provision output. Subsequent verification must not run that PHP: it selects the set from trusted escrow metadata using sorted fixture names (arguments, AI_PG_FIXTURES, or kb_docs canvas_home). If several keys match, selection fails closed with APG_ESCROW_NONE; use the known absolute escrow verifier path. AI_PG_BASELINE_REF can narrow selection. Baseline directory and ref overrides must match provision at escrow time. Container paths starting with /var/www/html in AI_PG_BASELINE_DIR are translated to the host project. The saved destinations are used for subsequent repairs.

For each attempt, the command contract is:

ddev ai-eval-verify kb_docs canvas_home --restore
ddev ai-eval-reset kb_docs canvas_home
# Run the agent and wait for it and its child processes to stop.
ddev ai-eval-run <task-id> # Creates .ai-proving-ground/runs/<run-id>/.
ddev ai-eval-verify kb_docs canvas_home
# Only on successful verification:
ddev ai-eval-grade <task-id> --run=<run-id>

A nonzero verify means do not grade; record TAMPERED and preserve stderr as evidence. Baseline-file and tag differences produce APG_ESCROW_MISMATCH; --restore repairs those and reports APG_ESCROW_RESTORED. Judging-code changes, including added and removed files, produce APG_TAMPERED <path> <change> and always fail, even with --restore. Code is never auto-restored. A successful check ends with APG_ESCROW_OK.

The ddev dispatcher and its host wrapper are themselves agent-writable. The wrapper executes the escrow verifier, but an attempt can replace the wrapper before dispatch. Therefore an unattended harness must save the absolute escrow path returned by setup in its own trusted state and invoke that verifier directly for both checks above, using a trusted host Bash executable:

/usr/bin/bash "$ESCROW_SET_DIR/escrow-verify.sh" "$BED_APPROOT" "$ESCROW_SET_DIR" --restore
# Reset, run agent, stop agent, then:
/usr/bin/bash "$ESCROW_SET_DIR/escrow-verify.sh" "$BED_APPROOT" "$ESCROW_SET_DIR"

Keep harness code, these paths, and its environment outside the project. The agent must be stopped before verification and remain stopped through grading; this is a point-in-time check, not protection against concurrent writes.

Escrow refuses APG_ESCROW_UNSAFE if its resolved directory is inside the project or under a recognized bind source in .ddev/docker-compose.*.yaml. It recognizes conventional short volume syntax and long source: syntax, absolute paths, relative paths based on .ddev, ${HOME}, $HOME, and ~. It does not implement a general YAML or Compose evaluator: YAML aliases, inline mapping syntax, generated Compose overrides, and environment-file interpolation are not resolved. Unresolved dollar substitutions in recognized sources fail closed. The operator must inspect effective mounts for other syntax and mounts configured elsewhere. The escrow is only as safe as the host account's own mounts and permissions. Use a directory outside every container mount.

This does not hide ground truth: the agent can still read it. The agent can change the database, which is the graded deliverable. Grade envelopes and runs/<id>/grade.json remain inside the project and can be rewritten after the fact. A harness must retain the grade stdout it captured itself in trusted storage, together with verification evidence. Hashes are integrity evidence relative to the trusted escrow, not signatures against a compromised host.

Run the host-only regression suite with bash tests/escrow.test.sh. It creates a temporary Git bed and puts a stub ddev executable on PATH that returns a fixed fixture key only for the setup exec drush php:script call. No ddev, Docker, or Drush process is launched by the suite.

drupal / ai_proving_ground

Package info

Security

Statistics