README

Evidence-aware risk review guardrails for Laravel applications, AI products, RAG systems, and MCP tools.

Official docs: https://doc.laravel-evidence-risk-review.padosoft.com

This package labels source strength, detects risky claims, keeps LLM calls default-OFF, records review evidence when enabled, and exposes the same core engine through PHP, Artisan, HTTP, and MCP surfaces.

Why It Exists
The Value It Adds
Features
What Is Inside
When The AI Steps In
Quick Start
PHP Surface
Artisan Surface
HTTP Surface
MCP Surface
Configuration
Profiles And Taxonomy
Review Logs
Eval Metric
Testing
Architecture
Security
Part Of The Padosoft AI Suite
Contributing
License

Why It Exists

LLM answers often look confident before they are well supported. padosoft/laravel-evidence-risk-review gives Laravel teams a deterministic review layer that can run before publishing, storing, streaming, or acting on AI-generated content.

The core idea is simple:

classify every source into a configured evidence tier
compare claim assertiveness against required evidence strength
run cheap deterministic checks first
call expensive or external LLM review only when explicitly enabled
return structured findings that adapters can use consistently

The Value It Adds

Most "AI safety" tooling either ships a heavyweight LLM judge that costs a token call on every request, or a regex blocklist that misses the real problem. This package sits in the middle and gives you the part that is hard to build well:

Cheaper by design. Deterministic checks run first and resolve most artifacts for free. The expensive LLM pass is the exception, not the default — and it only runs when you turn it on.
Auditable, not magical. Every result is a structured list of findings with the evidence tier, the claim, and the rule that fired. You can log it, diff it, and explain it to a compliance reviewer.
One engine, four doors. The exact same ReviewEngine is reachable from PHP, Artisan, HTTP, and MCP. No drift between your API, your CLI, and your agent tools.
Domain-aware out of the box. Ship-ready profiles for engineering, medical, legal, and finance encode "a definitive medical claim needs a peer-reviewed source, a blog does not count."
Truly standalone. Zero coupling to any host app, knowledge base, or LLM SDK. You bind your own LLM contract; the package never picks a vendor for you.
Safe defaults. HTTP, MCP, LLM, and persistence are all default-OFF. Installing it changes nothing until you opt in.

Features

✅ Evidence-tier labeling — classify each source (guideline, peer-reviewed, official, preprint, news, blog, search hint, unverified) with configurable rules.
✅ Risk sweep — deterministic checks compare claim assertiveness against required evidence strength and flag unsupported, overconfident, or contradicted claims.
✅ Default-OFF LLM review — optional second pass through a host-provided EvidenceReviewerLlmContract; no LLM SDK ships in the package.
✅ Five built-in profiles — default, engineering, medical, legal, finance, each tuning which checks run and what minimum tier each assertiveness level requires.
✅ Four surfaces, one engine — PHP facade, Artisan commands, default-OFF HTTP API (OpenAPI 3.1), and framework-agnostic MCP tool registry.
✅ Append-only review logs — null, array, or database stores for an immutable evidence trail.
✅ Zero-token eval metric — plug the engine into padosoft/eval-harness as evidence-risk: deterministic groundedness scoring with no provider call, so it runs on every row of every build next to the LLM judge you can only afford on some of them.
✅ Stable contracts — structured findings, a stable JSON error envelope, and deterministic Artisan exit codes for CI gating.
✅ Standalone & host-agnostic — no AskMyDocs, knowledge-base, or host-namespace dependency; enforced by architecture tests.

What Is Inside

Surface	Purpose
PHP service and facade	Direct package API for Laravel code.
Artisan commands	Local review, profile, taxonomy, and log inspection.
HTTP API	Default-OFF REST endpoints with OpenAPI 3.1.
MCP registry	Framework-agnostic tool definitions and handlers.
Review logs	Null, in-memory, and database append-only stores.
Profiles	Built-in default, engineering, medical, legal, and finance profiles.

When The AI Steps In

This package is not an LLM wrapper. By default it never calls a model — every review you saw above runs on pure, deterministic PHP. That keeps reviews fast, free, and reproducible.

The LLM pass exists for the cases deterministic rules cannot judge alone: nuanced claim-vs-source semantic alignment, paraphrase detection, or subtle contradictions. Here is the decision flow:

1. Label every source into an evidence tier            (deterministic, always)
2. Run the risk sweep: assertiveness vs required tier   (deterministic, always)
3. llm.enabled === true AND a contract is bound?
       no  -> return deterministic findings             (done, zero token cost)
       yes -> run the LLM reviewer for semantic checks   (only now is a model called)
4. Merge findings, optionally persist, return result

Turn it on only when you need it, and only after binding your own reviewer:

// In a service provider of the HOST app — the package ships no LLM SDK.
$this->app->bind(
    \Padosoft\EvidenceRiskReview\Contracts\EvidenceReviewerLlmContract::class,
    \App\Ai\MyEvidenceReviewer::class,
);

EVIDENCE_RISK_REVIEW_LLM_ENABLED=true

If llm.enabled is true but no contract is bound, the engine fails loudly instead of silently skipping the AI step.

Quick Start

Install the package:

composer require padosoft/laravel-evidence-risk-review

Publish config and the optional database migration:

php artisan vendor:publish --tag=evidence-risk-review-config
php artisan vendor:publish --tag=evidence-risk-review-migrations

Run a dry review from PHP:

use Padosoft\EvidenceRiskReview\Data\ReviewArtifact;
use Padosoft\EvidenceRiskReview\Facades\EvidenceRiskReview;

$result = EvidenceRiskReview::review(new ReviewArtifact(
    artifactId: 'answer-123',
    answerText: 'This likely helps when the documented prerequisites are met.',
));

return $result->toArray();

You get back a structured result you can inspect, log, or gate on — something like:

[
    'review_id'      => 'rev_...',
    'artifact_id'    => 'answer-123',
    'profile_key'    => 'default',
    'findings'       => [],        // each finding names the claim, the tier, and the rule that fired
    'claim_verdicts' => [],
    'source_tiers'   => [],
    'risk_score'     => 0,
    // ...budget, reviewed_at, metadata
]

Run the same review from the CLI:

php artisan evidence:review artifact.json --dry-run

The command exits 0 when there are no findings and 2 when findings exist, so it drops straight into a CI gate.

That is the whole loop. Enable nothing else until you need it — HTTP, MCP integrations, LLM calls, and persistence are all opt-in and stay off until you turn them on.

PHP Surface

use Padosoft\EvidenceRiskReview\Facades\EvidenceRiskReview;

$arrayResult = EvidenceRiskReview::reviewArray([
    'artifact_id' => 'answer-124',
    'answer_text' => 'This always cures the condition.',
    'claims' => [[
        'id' => 'c1',
        'text' => 'This always cures the condition.',
        'assertiveness' => 'definitive',
        'source_ids' => ['s1'],
    ]],
    'sources' => [[
        'id' => 's1',
        'declared_tier' => 'blog',
    ]],
    'options' => [
        'profile_key' => 'medical',
        'dry_run' => true,
    ],
]);

$tier = EvidenceRiskReview::labelTier([
    'id' => 'source-1',
    'url' => 'https://arxiv.org/abs/1234.5678',
]);

$profiles = EvidenceRiskReview::listProfiles();
$taxonomy = EvidenceRiskReview::taxonomy();

Artisan Surface

php artisan evidence:review artifact.json --dry-run
php artisan evidence:profiles
php artisan evidence:taxonomy
php artisan evidence:log --limit=25

evidence:review exits with:

Code	Meaning
`0`	Review completed and no findings were produced.
`2`	Review completed and findings were produced.
`1`	Invalid input, unknown profile, unavailable dependency, or runtime failure.

HTTP Surface

The HTTP API is default-OFF. Enable it explicitly:

'api' => [
    'enabled' => env('EVIDENCE_RISK_REVIEW_API_ENABLED', false),
    'prefix' => env('EVIDENCE_RISK_REVIEW_API_PREFIX', 'evidence-risk-review/api'),
    'middleware' => [],
],

Available endpoints when enabled:

POST /evidence-risk-review/api/reviews
GET  /evidence-risk-review/api/reviews            ?page=&per_page=&tenant=&profile=&min_verdict=
GET  /evidence-risk-review/api/reviews/{review}
GET  /evidence-risk-review/api/profiles
GET  /evidence-risk-review/api/profiles/{key}
GET  /evidence-risk-review/api/taxonomy
GET  /evidence-risk-review/api/openapi.yaml

GET /reviews (v1.1.0) returns a paginated review log — { data, current_page, last_page, per_page, total } where each row is { review_id, artifact_id, profile_key, max_verdict, risk_score, tenant_id, created_at }. It is backed by the database review-log store; with the null / array store it returns an empty (or in-memory) page. Filter by profile, min_verdict (keeps reviews whose highest-severity verdict is at least the given one), and tenant.

Multi-tenancy (v1.1.0)

The package is tenancy-agnostic but multi-tenant-safe. Bind a Padosoft\EvidenceRiskReview\Contracts\TenantResolver in a host that runs more than one tenant:

$this->app->singleton(
    \Padosoft\EvidenceRiskReview\Contracts\TenantResolver::class,
    fn () => new class implements \Padosoft\EvidenceRiskReview\Contracts\TenantResolver {
        public function current(): ?string { return app('your.tenant.context')->id(); }
    },
);

When a resolver is bound, every persisted review is stamped with the resolved tenant (the client payload can never spoof it), and the log read paths (GET /reviews + GET /reviews/{review}) are forced to the current tenant — so one tenant can never read or pollute another's reviews. The default NullTenantResolver preserves the original single-tenant behaviour.

HTTP errors use a stable envelope:

{
  "error": {
    "code": "validation_error",
    "message": "Expected non-empty string at [artifact_id].",
    "details": {}
  }
}

MCP Surface

The MCP layer is framework-agnostic:

use Padosoft\EvidenceRiskReview\Mcp\McpToolRegistry;

$registry = app(McpToolRegistry::class);

$definitions = array_map(
    static fn ($definition) => $definition->toArray(),
    $registry->definitions(),
);

$result = $registry->handle('evidence_review.assess', [
    'artifact_id' => 'answer-125',
    'answer_text' => 'No claims to check.',
    'options' => ['dry_run' => true],
]);

Available tools:

evidence_review.assess
evidence_review.label_tier
evidence_review.list_profiles

Configuration

The package config is published to config/evidence-risk-review.php.

Important defaults:

Key	Default	Effect
`api.enabled`	`false`	HTTP routes are not registered unless enabled.
`mcp.enabled`	`false`	Hosts decide if and how to expose MCP tools.
`llm.enabled`	`false`	No external LLM calls happen by default.
`review_log.store`	`null`	No persistence unless `array` or `database` is configured.
`default_profile`	`default`	Review profile used when no option is supplied.

See .env.example for the supported environment variables.

Profiles And Taxonomy

Built-in profiles:

default
engineering
medical
legal
finance

Evidence tiers are configurable. Built-ins include guideline, peer-reviewed, official, preprint, news, blog, search hint, and unverified.

Profiles decide which risk checks are enabled and what minimum source tier each claim assertiveness level requires.

Review Logs

Supported stores:

null: default, append is a no-op
array: useful for tests and in-process inspection
database: append-only table published through the package migration

Enable database logs:

EVIDENCE_RISK_REVIEW_LOG_STORE=database
EVIDENCE_RISK_REVIEW_LOG_CONNECTION=mysql
EVIDENCE_RISK_REVIEW_LOG_TABLE=evidence_risk_review_logs

Eval Metric

padosoft/eval-harness scores a pipeline against a golden dataset, and its most capable metric — llm-as-judge — is also its most expensive: a thousand rows at three repetitions is three thousand paid calls that exist purely to grade, by a grader that is itself a model and disagrees with itself between runs.

This engine answers a narrower question — is this answer actually supported by the sources it cites, and does its confidence match its evidence? — deterministically, with no provider call at all. So it costs nothing per row and can run on every row of every build, next to the judge you run on some of them.

composer require --dev padosoft/eval-harness

use Padosoft\EvidenceRiskReview\Eval\EvidenceRiskMetric;

$eval->dataset('rag.grounding')
    ->loadFromYaml(database_path('evals/rag.grounding.yaml'))
    ->withMetrics(['llm-as-judge', EvidenceRiskMetric::class])
    ->register();

The claims and sources live on the row, because only the dataset knows what the pipeline was supposed to have grounded itself in:

- id: refund-window
  input: { question: 'What is the refund window?' }
  expected_output: '30 days from delivery'
  metadata:
    evidence:
      profile: default
      claims:
        - { id: c1, text: 'Refunds are accepted for 30 days.', assertiveness: definitive, source_ids: [policy] }
      sources:
        - { id: policy, url: 'https://example.test/returns', declared_tier: official }

The metric scores 1 − risk_score, which puts the harness's 0.5 pass line between soften (0.67, passes) and flag_for_human_review (0.33, fails): a hedge-worthy overstatement is a note; an answer that needs a human is a failure. Pass minScore to make it binary instead.

Four behaviours worth knowing:

The score is a function of the answer, not just the annotation. Each declared claim is reviewed only if the produced answer actually asserted it, and a row whose answer asserted none of them scores 0.0 — otherwise the metric would be a constant per row and could not detect a regression at all.
No provider call, guaranteed (cheap_only: true, not merely label_via_llm: false): the engine otherwise runs its heavy LLM checks whenever the host has the integration enabled, which would have billed a provider on exactly the rows that were already failing.
A row with no metadata.evidence block scores 1.0 and says so in its details. Failing every un-annotated row would make the metric impossible to adopt on a dataset that already exists — but a block that is present and malformed raises, because treating it as un-annotated would let a broken row into the aggregate as a pass.
Reviews run during an eval never reach the audit log (dry_run: true). That log records what production did; a CI run is not production.

Full guide: Eval Metric.

Testing

Local gates:

composer validate --strict --no-interaction --no-ansi
vendor/bin/pint --test
vendor/bin/phpstan analyse --memory-limit=512M --no-progress
vendor/bin/phpunit
npx --yes @redocly/cli@latest lint resources/openapi.yaml

Live tests are opt-in and skip unless explicitly enabled:

EVIDENCE_RISK_REVIEW_LIVE=1 vendor/bin/phpunit --testsuite Live

Architecture

The package keeps one core engine and thin adapters:

ReviewArtifact / ReviewOptions
        |
        v
ReviewEngine
        |
        +-- EvidenceTierLabeler
        +-- RiskSweepEngine
        +-- EvidenceReviewerLlmContract
        +-- ReviewLogStore
        |
        v
PHP facade / Artisan / HTTP / MCP

Business rules live in core services and DTOs. Controllers, commands, and MCP handlers adapt input and output only.

Security

LLM calls are default-OFF.
HTTP routes are default-OFF.
Review logging is default-OFF.
Unknown config values fail loudly.
The package has no AskMyDocs or host-app namespace dependency.

Report vulnerabilities through the process in SECURITY.md.

Part Of The Padosoft AI Suite

padosoft/laravel-evidence-risk-review is one of the Padosoft AI sister packages — a family of standalone, host-agnostic Laravel building blocks for shipping trustworthy AI features. Each is independent, but they compose cleanly:

Package	What it does
padosoft/laravel-evidence-risk-review-admin	🤩 A gorgeous web admin panel for this package — browse reviews, findings, evidence tiers, and profiles from a cross-mounted SPA. The visual companion to the engine.
padosoft/laravel-ai-regolo	EU-based Regolo.ai provider adapter for `laravel/ai` (chat, streaming, embeddings, reranking).
padosoft/laravel-pii-redactor (+ `-admin`)	EU-grade field-level PII detection and masking inside the app boundary.
padosoft/laravel-flow (+ `-admin`)	Saga engine with approval gates, webhook outbox, and replay for AI workflows.
padosoft/eval-harness (+ `-ui`)	Golden datasets, RAG metrics, cohorts, adversarial testing, and LLM-as-judge regression gates.

These packages power lopadova/askmydocs, Padosoft's RAG platform — which integrates this evidence-tier and risk-review engine to surface low-confidence claims directly in its RAG prompts. If you want the full governed RAG stack rather than a single building block, start there.

Contributing

Read CONTRIBUTING.md, AGENTS.md, CLAUDE.md, docs/RULES.md, and docs/LESSON.md before opening a PR.

The repo includes a Claude/agent/vibe-coding pack under .claude/ and skills/ so future agent sessions inherit the project rules.

License

Apache-2.0. See LICENSE.

padosoft / laravel-evidence-risk-review

Maintainers

Package info

Statistics

Security

README

Table Of Contents

Why It Exists

The Value It Adds

Features

What Is Inside

When The AI Steps In

Quick Start

PHP Surface

Artisan Surface

HTTP Surface

Multi-tenancy (v1.1.0)

MCP Surface

Configuration

Profiles And Taxonomy

Review Logs

Eval Metric

Testing

Architecture

Security

Part Of The Padosoft AI Suite

Contributing

License