insalyon/acme-plugin-etl

ACME Plugin to create ETL Job

v1.8.0 2026-03-20 14:50 UTC

Requires

• php: >=8.3
• insalyon/acme: v2.2.01

CC-BY-NC-4.0

• Charbel FOUREL
• Johan MAURICE
• Lucas DUPUY
• Aurelien Leygues

README

A synchronization plugin that runs ETL jobs between two heterogeneous data sources.

Each job reads data from an origin layer, compares it with a target layer, and applies the necessary CRUD operations to keep them in sync.

flowchart TB
    subgraph ACME_ETL
        JOB_NAME_01([JOB_NAME_01])
        JOB_NAME_02([JOB_NAME_02])
    end
    subgraph DATA_SOURCE
        ORIGIN_LAYER[(ORIGIN_LAYER<p>postgresql, oracle, mysql, api http, ldap</p>)]
        TARGET_LAYER[(TARGET_LAYER<p>postgresql, oracle, mysql, api http, ldap</p>)]
    end
    ORIGIN_LAYER --> ACME_ETL --> TARGET_LAYER
    CLIENT --GET http://cname/api/v1/synch:job:name:01--> JOB_NAME_01
    CLIENT --GET http://cname/api/v1/synch:job:name:02--> JOB_NAME_02

ETL Pipeline

For each job, the plugin executes the following pipeline:

// Hook executed before anything else
$this->preProcess();

// Fetch data from the origin data source
$originDatas = $this->extractOriginEntity();

// Fetch data from the target data source
$targetDatas = $this->extractTargetEntity();

// Normalize and index origin data to make it comparable with target
$originDatas = $this->transformOriginToTarget($originDatas);

// Normalize and index target data
$targetDatas = $this->transformTarget($targetDatas);

// Diff origin vs target to generate CRUD operations
$generatedRequest = $this->generateRequestAfterCompare($originDatas, $targetDatas);

// Execute generated requests against the target layer (HTTP, SQL, or LDAP)

Plugin implementation: ACMEPluginETL.php

Known Limitations

• Only the SQL and HTTP layers can be configured independently between origin and target. The LDAP layer is shared — you cannot define a custom LDAP connection in addition to the default one.

Documentation

Topic	Link
Quick start & configuration	QuickStart.md
System requirements	QuickStart.md
Full parameter reference	Configuration.md
Origin layer parameters	ACME-ETL-Params-ORIGIN_LAYER.md
Target layer parameters	ACME-ETL-Params-TARGET_LAYER.md
Changelog	changelog.md
ACME core project	ACME

Monitoring

When the ACME metrics layer is enabled in cfg.ini, the plugin automatically pushes runtime metrics to the configured metrics database.

A Grafana dashboard definition is available for visualization (requires plugin version >= v1.6.5-alpha).