README

ai-integration-tester verifies AI provider integrations through reusable conformance scenarios. It exercises chat responses, tool calls, typed scenario results, and provider-specific behavior through a small CLI application.

Requirements

PHP 8.3 or later
Composer
adachsoft/ai-integration ^0.6, ^0.7, or ^0.8
API credentials for each provider used by production conformance tests or live examples

Installation

Install the package as a development dependency:

composer require --dev adachsoft/ai-integration-tester

The repository includes a typed configuration file at .ai-test.php. It registers the built-in OpenAI and Deepseek profiles, the default provider, and the tools used by the conformance scenarios. Configuration errors are reported instead of silently falling back to defaults.

Provider credentials

The default environment reader loads a project-root .env file when needed. Set the credentials for the providers you intend to use:

OPENAI_API_KEY=your-openai-api-key
DEEPSEEK_API_KEY=your-deepseek-api-key

The built-in profiles use gpt-4o-mini for OpenAI and deepseek-chat for Deepseek. Production conformance tests make real provider requests when credentials are available. Tests for a provider are skipped with an explanatory message when that provider's API key is not configured.

Custom provider conformance cases

Provider-specific test suites can extend AdachSoft\AiIntegrationTester\Conformance\AbstractProviderConformanceTestCase. Override createScenarioCollection() to provide provider-specific scenarios, getRequiredTools() to register their tools, and getAgentProvider() when the tests need a custom agent provider. The default implementations supply the built-in scenarios, tools, and agent provider.

Custom agent providers

Typed configuration accepts an optional AgentProviderInterface as the agentProvider argument of TesterConfigDto. The configured provider supplies the AgentSimulatorInterface used by CLI scenario runs. When omitted, the CLI creates the default agent simulator from the selected provider profile.

return new TesterConfigDto(
    providerProfiles: $providerProfiles,
    defaultProviderId: $defaultProviderId,
    tools: $tools,
    agentProvider: new MyAgentProvider(),
);

MyAgentProvider must implement AdachSoft\AiIntegrationTester\Agent\AgentProviderInterface and return an AdachSoft\AiIntegrationTester\Simulator\AgentSimulatorInterface from getAgent().

Generation parameters

SessionConfigDto accepts optional GenerationParametersDto. Omit the third constructor argument to send no explicit generation parameters and let the provider or model use its defaults; pass a DTO when a scenario requires specific generation settings. The built-in conformance scenarios use provider defaults.

CLI

List registered scenarios without connecting to a provider:

php bin/ai-test scenario:list

Run a scenario with a selected provider:

php bin/ai-test scenario:run tool_calling_conformance --provider=openai
php bin/ai-test scenario:run tool_calling_conformance --provider=deepseek

Validate a provider by running the registered smoke scenario, or inspect the configured tool contracts:

php bin/ai-test provider:validate --provider=openai
php bin/ai-test tool:contract

The default provider is configured in .ai-test.php; --provider=<id> overrides it for a command. Scenario and provider setup is performed when a command needs it, so listing scenarios does not require API credentials.

Examples

The scripts in examples/ invoke the same CLI and return its exit code:

php examples/conversation_openai.php
php examples/conversation_deepseek.php
php examples/time_and_file_conversation.php           # defaults to Deepseek
php examples/time_and_file_conversation.php openai
php examples/time_and_file_conversation.php deepseek

The time-and-file example runs the time_and_file_conversation scenario, which exercises the current-time and virtual-file tools. Configure the corresponding provider key in .env before running a live example.

Tests and quality checks

Run the local unit suite and the separate live-provider suite:

vendor/bin/phpunit --testsuite unit
vendor/bin/phpunit --testsuite production

The test suite supports PHPUnit 12.4 and 13.x. The production suite makes live requests for providers with configured API keys; conformance tests for providers without a configured key are skipped with an explanatory message. Additional project checks are:

vendor/bin/phpstan analyse
composer cs:check

Project layout

src/ — provider profiles, configuration, CLI, simulators, conformance scenarios, assertions, and typed results.
tests/Unit/ — isolated unit tests.
tests/Production/ — live provider conformance tests.
examples/ — runnable CLI examples.

adachsoft / ai-integration-tester

Package info

Security

Statistics