README

Framework primitives for AskMyDocs connectors — write a Laravel package, plug it into any AskMyDocs instance.
Implement ConnectorInterface on your favourite data source (Google Drive, Notion, Confluence, a CSV bucket, an internal API, ...) and let AskMyDocs ingest it as RAG-grounded knowledge with OAuth, encrypted-at-rest credentials, retry-aware queued syncs, per-tenant isolation, and a cadence scheduler — all wired automatically by composer discovery.

Why this package
What you get
Architecture at a glance
Installation
Quick start — write your first connector in 50 lines
The 10-method contract
Optional: credential form interface
How auto-discovery works
Credential vault — encrypted, atomic, tenant-scoped
Scheduler + sync job
Multi-tenancy (R30 + R31)
Configuration reference
Testing
Roadmap
License

Why this package

AskMyDocs is an enterprise-grade RAG + canonical knowledge compilation system. Out of the box it ingests markdown from disk, the chat UI, an HTTP API, and a Git-driven workflow.

But the knowledge people actually want to query lives in Google Drive, Notion, Confluence, Jira, OneDrive, Evernote, Fabric, Slack, Salesforce, HubSpot, a private S3 bucket, a custom CRM — anywhere except markdown-on-disk.

This package is the smallest possible surface for shipping a new connector:

A 10-method ConnectorInterface you implement.
A BaseConnector that gives you OAuth state-token CSRF, credential refresh, and tenant-scoped installation lookup for free.
A registry that auto-discovers your package the moment somebody composer requires it — zero config edits on the consumer side.
An OAuthCredentialVault that handles encryption-at-rest, refresh-token semantics, and atomic concurrent writes (no read-modify-write data loss on shared cursor blobs).
A queued ConnectorSyncJob with exponential backoff, tenant restore, and failure-state recording.
A cadence scheduler that walks active installations every minute and dispatches due syncs.
Per-tenant isolation baked into every query (R30 / R31 — see Multi-tenancy).

Write the connector. Ship the package. Composer-require it from any AskMyDocs install. Done.

What you get

Surface	Class	What it does
Contract	`ConnectorInterface`	10 methods every connector implements
Base	`BaseConnector`	OAuth state-token CSRF, refresh helper, tenant lookup
Registry	`ConnectorRegistry`	Boot-time R23 validation + composer-extra auto-discovery
Vault	`Auth\OAuthCredentialVault`	AES-encrypted tokens, atomic `setExtraKey` (R21), tenant scope
Scheduler	`Scheduling\SyncScheduler`	Cadence walker, `chunkById(100)`, active-only filter
Job	`ConnectorSyncJob`	`$tries=3`, exponential backoff, tenant-restore safety
Models	`ConnectorInstallation` + `ConnectorCredential`	`BelongsToTenant` trait, cascade delete
Migrations	`connector_installations` + `connector_credentials`	Auto-loaded by the service provider
Exceptions	`ConnectorAuthException`, `ConnectorApiException`, `ConnectorPaginationLimitException`, `RegistryConfigurationException`	Distinct failure semantics: auth = no retry, api = retry, paginator-limit = partial success
DTOs	`SyncResult`, `HealthStatus`	Immutable outcomes
Tenancy	`Support\TenantContext` + `Models\Concerns\BelongsToTenant`	Request-scoped tenant, auto-fill on creating
Credential form (optional)	`Contracts\SupportsCredentialForm` + `Support\CredentialField`	Opt-in interface for credential-based connectors (IMAP, API key, …) — host renders a native admin form instead of OAuth redirect
Folder discovery (optional, v1.4)	`Contracts\SupportsFolderDiscovery`	Opt-in interface — `listAvailableFolders()` enumerates the live containers (IMAP folders, labels, spaces) an operator can whitelist; the connector owns auth + client lifecycle
Provenance (optional, v1.5)	`Contracts\DeclaresProvenance` + `ProvenanceTier`	Opt-in interface — `provenanceTier(int $installationId)` declares who authored the content this installation ingests, so the host can record it per document. Not a curation tier
Connection settings (optional, v1.4)	`Contracts\SupportsConnectionSettings` + `Support\CredentialField`	Opt-in interface — `connectionSettingsSchema()` declares the editable post-install sync knobs (window, folders, filters), rendered by the host as a generic settings editor

Architecture at a glance

                         ┌──────────────────────────┐
                         │  Your connector package  │
                         │  composer extra.askmydocs│
                         └─────────────┬────────────┘
                                       │  auto-discovered
                                       ▼
┌────────────────────┐         ┌──────────────────┐         ┌──────────────────────────┐
│ Cadence scheduler  │ ──────▶ │ ConnectorRegistry│ ◀────── │ Host: config/connectors  │
│ (every minute)     │         │ R23 boot-time    │         │ ::built_in (optional)    │
└────────┬───────────┘         │ FQCN validation  │         └──────────────────────────┘
         │                     └─────────┬────────┘
         │ dispatch                      │ resolve by key()
         ▼                               ▼
┌────────────────────┐         ┌──────────────────┐         ┌──────────────────────────┐
│ ConnectorSyncJob   │ ──────▶ │ Your connector   │ ──────▶ │ OAuthCredentialVault     │
│ tenant-restore     │         │ syncIncremental()│         │ AES + lockForUpdate (R21)│
│ tries=3, backoff   │         └─────────┬────────┘         └──────────────────────────┘
└────────────────────┘                   │
                                         │ fetches changed docs
                                         ▼
                              ┌────────────────────────┐
                              │ Host ingest pipeline   │
                              │ (e.g. AskMyDocs        │
                              │  IngestDocumentJob)    │
                              └────────────────────────┘

Installation

composer require padosoft/askmydocs-connector-base

The service provider is auto-discovered (Laravel package discovery). The package ships its own migrations — run them:

php artisan migrate

Want to copy the migrations into your host app's database/migrations/ (e.g. to tweak tenant_id length)? Publish them:

php artisan vendor:publish --tag=connector-migrations

Same for the config:

php artisan vendor:publish --tag=connector-config

Wire the scheduler from your host app's bootstrap/app.php:

use Padosoft\AskMyDocsConnectorBase\Scheduling\SyncScheduler;

->withSchedule(function (Schedule $schedule): void {
    (new SyncScheduler)->registerSchedules($schedule);
})

That's it. Connector packages installed via composer are now auto-discovered and synced on cadence.

Quick start — write your first connector in 50 lines

Create a new Laravel package. Add padosoft/askmydocs-connector-base to its require. Declare your connector class FQCN under extra.askmydocs.connectors:

// composer.json (your package)
{
    "name": "you/askmydocs-connector-myapi",
    "require": {
        "padosoft/askmydocs-connector-base": "^1.0"
    },
    "autoload": {
        "psr-4": { "You\\AskMyDocsConnectorMyApi\\": "src/" }
    },
    "extra": {
        "askmydocs": {
            "connectors": [
                "You\\AskMyDocsConnectorMyApi\\MyApiConnector"
            ]
        }
    }
}

Implement the connector:

namespace You\AskMyDocsConnectorMyApi;

use Carbon\Carbon;
use Illuminate\Http\Request;
use Padosoft\AskMyDocsConnectorBase\BaseConnector;
use Padosoft\AskMyDocsConnectorBase\HealthStatus;
use Padosoft\AskMyDocsConnectorBase\SyncResult;
use Padosoft\AskMyDocsConnectorBase\Exceptions\ConnectorAuthException;

final class MyApiConnector extends BaseConnector
{
    public function key(): string         { return 'my-api'; }
    public function displayName(): string { return 'My API'; }

    public function oauthScopes(): array
    {
        return ['read:docs'];
    }

    public function initiateOAuth(int $installationId): string
    {
        $state = $this->issueOAuthState($installationId);
        return 'https://my-api.example.com/oauth/authorize?state='.$state.'&...';
    }

    public function handleOAuthCallback(int $installationId, Request $request): void
    {
        if (! $this->consumeOAuthState($installationId, (string) $request->query('state'))) {
            throw new ConnectorAuthException('Bad state');
        }
        // Exchange code -> token, then:
        $this->vault->setCredentials($installationId, 'access-token', refreshToken: 'refresh');
    }

    public function syncFull(int $installationId): SyncResult
    {
        return $this->syncIncremental($installationId, null);
    }

    public function syncIncremental(int $installationId, ?Carbon $since): SyncResult
    {
        // Fetch changed docs, dispatch host ingest jobs, count them.
        return new SyncResult(
            documentsAdded: 5,
            documentsUpdated: 2,
            documentsRemoved: 0,
            errors: [],
            completedAt: Carbon::now(),
        );
    }

    public function disconnect(int $installationId): void
    {
        $this->vault->clearCredentials($installationId);
    }

    public function health(int $installationId): HealthStatus
    {
        return HealthStatus::healthy();
    }
}

composer require you/askmydocs-connector-myapi in any AskMyDocs install — the registry auto-discovers it, the scheduler starts dispatching it on cadence, the admin UI lists it in the available-connectors picker.

The 10-method contract

Every connector implements 10 methods (3 metadata + 1 scope + 2 OAuth + 2 sync + 1 disconnect + 1 health):

Method	Purpose	Throws
`key(): string`	Stable kebab-case identifier (`google-drive`, `notion`). Used as URL slug + `connector_installations.connector_name`.	—
`displayName(): string`	Human label shown in the admin UI.	—
`iconUrl(): string`	Connector logo URL. `BaseConnector` provides a default that resolves `public/connectors/{key}.svg` via `asset()`.	—
`oauthScopes(): array`	List of scope strings the provider requires. Surfaced to the user in the install confirmation dialog.	—
`initiateOAuth(int): string`	Build the provider's authorization URL. Use `$this->issueOAuthState()` for CSRF.	`ConnectorAuthException`
`handleOAuthCallback(int, Request): void`	Exchange auth code -> tokens, verify state, persist via `$this->vault->setCredentials()`.	`ConnectorAuthException` on any failure
`syncFull(int): SyncResult`	Full discovery + ingestion. Long-running. Called at install + operator re-sync.	propagates upstream errors
`syncIncremental(int, ?Carbon): SyncResult`	Delta since `$since`. Falls back to `syncFull` when `$since === null`. Called by the cadence scheduler.	`ConnectorApiException` for transient (retry), `ConnectorAuthException` for credentials (no retry)
`disconnect(int): void`	Clear credentials, optionally revoke at provider.	swallow / log; framework deletes installation row after
`health(int): HealthStatus`	Fast (under 2s) side-effect-free probe.	returns `HealthStatus::errored(...)` instead of throwing

Optional: credential form interface

For connectors that use credentials instead of OAuth (IMAP, SMTP, API-key-based providers, ...), implement the optional SupportsCredentialForm interface alongside ConnectorInterface.

The host detects the interface via instanceof at install time and renders a native admin form. Each field is described by a CredentialField value object — call toArray() on each to produce the JSON shape the host expects.

use Padosoft\AskMyDocsConnectorBase\Contracts\SupportsCredentialForm;
use Padosoft\AskMyDocsConnectorBase\Support\CredentialField;

final class ImapConnector extends BaseConnector implements SupportsCredentialForm
{
    public function credentialFormSchema(): array
    {
        return [
            (new CredentialField(
                name: 'host', label: 'IMAP Host', type: 'text', target: 'connection', required: true,
            ))->toArray(),
            (new CredentialField(
                name: 'port', label: 'Port', type: 'number', target: 'connection', required: true, default: 993,
            ))->toArray(),
            (new CredentialField(
                name: 'username', label: 'Username', type: 'text', target: 'connection', required: true,
            ))->toArray(),
            (new CredentialField(
                name: 'password', label: 'Password', type: 'password', target: 'secret',
                required: true, secret: true,  // routed to vault, never stored in config_json
            ))->toArray(),
        ];
    }
}

CredentialField properties:

Property	Type	Description
`name`	`string`	Form-data key (e.g. `'host'`)
`label`	`string`	Human-readable UI label
`type`	`string`	`text` \| `number` \| `password` \| `select` \| `checkbox` \| `multiselect` (v1.4) \| `tags` (v1.4)
`target`	`string`	`connection` \| `config` → `config_json`; `auth_mode`; `provider`; `secret` → vault
`required`	`bool`	Whether the field must be filled
`secret`	`bool`	Masked in UI; routed to vault, never `config_json`
`default`	`mixed`	Pre-filled value (a list for `multiselect`/`tags`)
`options`	`array<string,string>`	For `select`/`multiselect` with a fixed set: `['value' => 'Label']`
`showIf`	`array{field:string,equals:string}\|null`	Conditional display rule
`help`	`string\|null`	Helper text rendered below the field
`group`	`string\|null`	Optional UI section heading
`discovery` (v1.4)	`string\|null`	For a live `multiselect`: names the discovery source (`'folders'` → `SupportsFolderDiscovery`)

Connectors that use only the standard OAuth redirect do not implement this interface — it is entirely opt-in and backward compatible.

Optional: folder discovery + editable settings (v1.4)

A connector that has selectable containers (IMAP folders, Gmail labels, …) and tunable sync behaviour opts into two more capabilities. The host detects each via instanceof (R23 — no connector-name branch) and renders a generic, schema-driven settings editor seeded with the installation's current config_json — no bespoke per-connector form.

use Padosoft\AskMyDocsConnectorBase\Contracts\SupportsConnectionSettings;
use Padosoft\AskMyDocsConnectorBase\Contracts\SupportsFolderDiscovery;
use Padosoft\AskMyDocsConnectorBase\Support\CredentialField;

final class ImapConnector extends BaseConnector implements
    SupportsCredentialForm,
    SupportsFolderDiscovery,
    SupportsConnectionSettings
{
    /** Live container list — the connector owns auth + client lifecycle. */
    public function listAvailableFolders(int $installationId): array
    {
        $client = $this->makeClient($installationId);   // handles basic + xoauth2 token refresh
        try {
            return $client->listMailboxes();             // list<string>, verbatim
        } finally {
            $client->close();
        }
    }

    /** Editable post-install sync knobs — every field target='config', never secret. */
    public function connectionSettingsSchema(): array
    {
        return [
            (new CredentialField(
                name: 'folders.include', label: 'Folders to sync', type: 'multiselect',
                target: 'config', default: [], group: 'Folders', discovery: 'folders',
                help: 'Empty = sync every non-excluded folder.',
            ))->toArray(),
            (new CredentialField(
                name: 'folders.exclude', label: 'Folders to skip', type: 'multiselect',
                target: 'config', default: [], group: 'Folders', discovery: 'folders',
            ))->toArray(),
            (new CredentialField(
                name: 'date_window_days', label: 'Sync window (days)', type: 'number',
                target: 'config', default: 365, group: 'Sync window', help: '0 = all history.',
            ))->toArray(),
            (new CredentialField(
                name: 'senders.exclude', label: 'Exclude senders', type: 'tags',
                target: 'config', default: [], group: 'Filtering',
            ))->toArray(),
        ];
    }
}

The settings field name is a dotted path the host writes into config_json (folders.include → config_json['folders']['include']) — exactly what the connector reads back at sync time, so a picked value round-trips 1:1. Both capabilities are opt-in and backward compatible.

Optional: declaring provenance (v1.5)

Ingestion records what a document is — its title, its path, its mime type — and nothing about where the authority of its text comes from. That gap is invisible until you notice the two very different things a connector can be doing:

a Drive or Confluence connector carries text written inside the organisation;
an IMAP connector carries text written by anyone who can send an email.

Both become retrieval grounding, and on a platform that also exposes tools to an agent, the second is attacker-reachable text arriving in a tool-calling context. Storing them as the same fact means no deployment can answer "how much of our corpus is externally authored?" — let alone act on the answer.

DeclaresProvenance closes that gap with a single method:

use Padosoft\AskMyDocsConnectorBase\Contracts\DeclaresProvenance;
use Padosoft\AskMyDocsConnectorBase\ProvenanceTier;

final class ImapConnector extends BaseConnector implements DeclaresProvenance
{
    public function provenanceTier(int $installationId): ProvenanceTier
    {
        // A mailbox accepts mail from anyone who knows the address.
        return ProvenanceTier::UntrustedExternal;
    }
}

Tier	Value	Means
`TrustedInternal`	`trusted-internal`	Written inside the organisation, through a system it controls
`UntrustedExternal`	`untrusted-external`	Written outside the organisation's control, or by an unverified author
`MachineGenerated`	`machine-generated`	Produced by a model or automated process rather than a person

The connector declares it, never the host. Only the fetcher knows whether a mailbox is an internal distribution list or a public contact address; inferring it host-side would be a heuristic over a fact the connector already had.

Resolved per installation, exactly like SupportsFolderDiscovery::listAvailableFolders(). Two installations of one connector routinely differ - an internal distribution list and a public contact address are the same IMAP code against sources with opposite authorship models. ConnectorRegistry keeps one instance per connector key, so the installation id has to be an argument; a zero-argument method would force ambient mutable state that is not set when the host resolves the tier on the ingestion path.

Opt-in and backward compatible. A connector that does not implement the interface keeps its exact current meaning: the host falls back to ProvenanceTier::default() — TrustedInternal, the tier every pre-existing connector already carried in practice. The ingestion contract signature does not change, so nothing built on the public connector template breaks.

The default is deliberately not the most cautious value. Defaulting to "untrusted" would relabel an entire existing corpus overnight and make the label mean nothing; external content is the exception a connector has to declare.

This is not a curation tier. A curation ranking answers "has a human vouched for this?". Provenance answers "who wrote it?". A page a human reviewed and accepted, summarising an external email, is fully curated and externally authored at once — collapsing the two loses the half that matters for trust.

Reading a stored value never throws, and the two failure modes get different answers:

null is a known absence - no declaration, i.e. every document written before this existed. It reads as default().
an unrecognised string is a tier this version does not understand, written by a newer one during a mixed deployment or after a rollback. It fails closed to UntrustedExternal.

Collapsing the second into the trusted default would invert the protection: a future tier meant to be more restrictive would read as safe on the older node, and isExternallyAuthored() would answer false for content nobody vouched for.

The tier is a label. This release enforces nothing with it — that is the point: enforcement is testable only against a corpus that is already labelled.

How auto-discovery works

ConnectorRegistry merges two sources at boot:

config/connectors.php::built_in — FQCN list for connectors the host app wires by hand (rare).
composer.lock packages — every entry whose extra.askmydocs.connectors is a non-empty array of FQCNs.

Each FQCN is resolved through the container and instanceof-checked against ConnectorInterface (R23). Failure modes:

Class missing -> RegistryConfigurationException: '...' does not exist
Class exists but doesn't implement -> RegistryConfigurationException: '...' does not implement ConnectorInterface
Two connectors return the same key() -> RegistryConfigurationException: Duplicate connector key '...'
Container can't instantiate -> RegistryConfigurationException: '...' could not be resolved

All boot-time. No silent fallthrough to a confusing "undefined method" later.

Credential vault — encrypted, atomic, tenant-scoped

OAuthCredentialVault is the single chokepoint for every connector's tokens:

AES-256 encryption at rest via Laravel Crypt. The DB row never sees plaintext.
Tenant-scoped reads — every query joins to connector_installations and filters by the active TenantContext. Cross-tenant reads return null, not the wrong tenant's tokens.
Refresh-aware — getAccessToken() returns null for expired tokens. Connectors call getRefreshToken() to rotate via the provider's /oauth2/token endpoint, then setCredentials() to persist the rotated pair.
R21 — atomic setExtraKey — concurrent writers updating different keys in extra_json (e.g. one connector storing bot_id, another storing changes_page_token) MUST NOT race. Implementation:

DB::transaction(function () use (...) {
    $row = ConnectorCredential::query()
        ->where(...)
        ->lockForUpdate()  // SELECT ... FOR UPDATE
        ->first();

    if ($row === null) {
        throw new ConnectorAuthException('credential row was deleted concurrently');
    }

    $extra = $row->extra_json ?? [];
    $extra[$key] = $value;
    $row->extra_json = $extra;
    $row->save();  // same transaction
});

A read-modify-write without the lock loses siblings under contention. The package was extracted from AskMyDocs precisely after this race was caught + fixed in production.

Scheduler + sync job

SyncScheduler::registerSchedules($schedule) registers one everyMinute() closure. The closure walks every STATUS_ACTIVE installation in chunkById(100) and dispatches ConnectorSyncJob for each that's due (i.e. last_sync_at + cadenceMinutes <= now()).

ConnectorSyncJob:

$tries = 3, $backoff = [60, 300, 900] — three attempts at 1m / 5m / 15m spacing.
$timeout = 600 — 10 min hard ceiling.
Tenant restore in finally — the job sets TenantContext to the dispatching tenant on entry, restores the prior value on exit. Long-lived queue workers handling jobs back-to-back for different tenants are R30-safe.
Status guards — non-ACTIVE installations short-circuit. De-registered connectors flip to STATUS_ERRORED with a clear message.
Failure semantics — ConnectorAuthException marks errored and exhausts retries (no point retrying bad credentials). ConnectorApiException and other throwables fail-and-retry per the backoff.

Multi-tenancy (R30 + R31)

Every model uses the BelongsToTenant trait:

R31 (write-side) — tenant_id auto-fills from TenantContext::current() on creating unless the caller passes an explicit value.
R30 (read-side) — forTenant($id) scope for explicit query scoping. Two tenants legitimately install the same connector under different tenant_ids — the composite UNIQUE (tenant_id, connector_name, label) makes the row pair structurally legal.

Host applications with their own TenantContext rebind via a container alias — both surfaces observe the same active tenant.

Multi-account & project binding (v1.3+)

A connector accepts more than one installation per (tenant_id, connector_name), disambiguated by a label (e.g. "support", "sales") — connect two IMAP mailboxes, several Drive/OneDrive accounts, or multiple Notion workspaces side by side. The composite UNIQUE is (tenant_id, connector_name, label); label defaults to 'default' so pre-v1.3 single-account rows upgrade transparently.

Each installation may bind to a real KB project_key (nullable column, indexed (tenant_id, project_key)). Resolve the effective project with the shared helper rather than re-deriving it per connector:

// inside your connector's syncFull()/syncIncremental()
$installation = $this->loadInstallation($installationId);
$projectKey = $this->resolveProjectKey($installation);
// → $installation->project_key, else config('kb.ingest.default_project'), else 'default'

Configuration reference

// config/connectors.php (publishable with --tag=connector-config)
return [
    'built_in' => [
        // \App\Connectors\BuiltIn\MyHostConnector::class,
    ],
    'default_sync_cadence_minutes' => env('CONNECTOR_DEFAULT_SYNC_CADENCE_MINUTES', 15),
    'per_connector_cadence' => [
        // 'google-drive' => 10,
        // 'notion'       => 30,
    ],
    'oauth_state_ttl_seconds' => env('CONNECTOR_OAUTH_STATE_TTL_SECONDS', 600),
    'sync_job_queue' => env('CONNECTOR_SYNC_JOB_QUEUE', 'default'),
    'providers' => [
        // Per-connector packages merge their own block here from their
        // own service providers via mergeConfigFrom().
    ],
];

Testing

composer install
vendor/bin/phpunit

Tests use Orchestra Testbench with SQLite in-memory. The default suite has zero external dependencies — every Laravel facade is in scope, every Crypt::encryptString() call uses a per-test APP_KEY, every model uses RefreshDatabase.

For connector packages built ON TOP of this base, follow the standard padosoft testing pattern: a default tests/Unit/ suite that uses Http::fake() (zero cost, runs in CI), plus an opt-in tests/Live/ suite that hits the real provider API (skipped when the env var is missing, invoked explicitly by maintainers).

Roadmap

v1.1 — Optional ChunkerInterface re-export once the AskMyDocs chunker value-object surface stabilises, so per-connector packages can ship provider-specific chunkers (already used in AskMyDocs for ConfluencePageChunker, JiraIssueChunker, AtomicNoteChunker).
v1.2 — Optional admin-trail helpers (audit event emission, PII redaction at the ingest boundary) lifted from AskMyDocs' host-side BaseConnector subclass into an AuditableBaseConnector mixin for hosts that want them out of the box.
v2.0 — MCPConnectorInterface companion for chat-time tool registration (Model Context Protocol). Connectors register tools the agent calls during a chat turn, complementing today's batch-sync model. Tracks the v4.5+ AskMyDocs agentic roadmap.

Community PRs welcome — open an issue first to discuss scope.

License

Apache-2.0 (c) Padosoft / Lorenzo Padovani. See LICENSE.

padosoft / askmydocs-connector-base

Maintainers

Package info

Statistics

Security