README

The Php etl bundle allows the usage of Oliver's PHP Etl library in symfony.

You should also check the PHP ETL's Easy Admin Bundle to have interfaces.

Installation

Install using composer
in /config/ create a directory etl
Enable bundle:

    \Oliverde8\PhpEtlBundle\Oliverde8PhpEtlBundle::class => ['all' => true],

Optional: Enable queue if you wish to allow users from the easy admin panel to do executions.

framework:
  messenger:
    routing:
        "Oliverde8\PhpEtlBundle\Message\EtlExecutionMessage": async

Optional: Enable creation of individual files for each log by editing the monolog.yaml

etl:
    type: service
    id: Oliverde8\PhpEtlBundle\Services\ChainExecutionLogger
    level: debug
    channels: ["!event"]

Usage

Creating an ETL chain

First read the documentation of the PHP ETL

Each chain is declare in a single file. The name of the chain is the name of the file created in /config/etl/. Example:

chain:
  "Dummy Step":
    operation: rule-engine-transformer
    options:
      add: true
      columns:
        test:
          rules:
            - get : {field: [0, 'uid']}

Executing a chain

./bin/console etl:execute demo '[["test1"],["test2"]]' '{"opt1": "val1"}'

The first argument is the input, depending on your chain it can be empty. The second are parameters that will be available in the context of each link in the chain.

Additional commands

Get a definition

./bin/console etl:get-definition demo

Observability — live execution graph

The bundle exposes a framework-agnostic live execution graph: the chain's topology plus per-operation state (items in/out, time, async in flight) and a streaming log tail. It is designed to be reused by any Symfony frontend (EasyAdmin, Sylius, a custom admin) — all the logic lives here, the frontend only mounts routes, loads the assets and renders one Twig partial.

How it works

Graph\ChainGraphBuilder turns a chain processor into a {nodes, edges} topology.
Graph\RunStateNormalizer turns the persisted/live OperationState into a node-keyed state map (both use the same dotted-path node ids, incl. split branches).
Controller\ExecutionObservabilityController serves three read-only JSON endpoints (guarded by EtlExecutionVoter::VIEW):
- GET .../etl/executions/{id}/graph — topology + last persisted state
- GET .../etl/executions/{id}/state — latest run-state (poll fallback)
- GET .../etl/executions/{id}/logs?offset= — incremental log tail
Resources/public/{js,css} ship a dependency-light Cytoscape widget (Cytoscape and dagre are vendored under Resources/public/vendor), and @Oliverde8PhpEtl/observability/graph.html.twig renders the container.

Wiring it into a frontend

Mount the routes (any prefix; put them behind your admin firewall):

# config/routes/oliverde8_etl.yaml
oliverde8_php_etl_observability:
    resource: '@Oliverde8PhpEtlBundle/Controller/'
    type: attribute
    prefix: /admin

Publish the assets: bin/console assets:install public.

Load the assets on the page that shows the graph and render the partial:

<link rel="stylesheet" href="/bundles/oliverde8phpetl/css/execution-graph.css">
<script src="/bundles/oliverde8phpetl/vendor/cytoscape.min.js"></script>
<script src="/bundles/oliverde8phpetl/vendor/dagre.min.js"></script>
<script src="/bundles/oliverde8phpetl/vendor/cytoscape-dagre.min.js"></script>
<script src="/bundles/oliverde8phpetl/js/execution-graph.js"></script>

{% include '@Oliverde8PhpEtl/observability/graph.html.twig' with { execution: execution } %}

(The EasyAdmin bundle does exactly this for you on the execution detail page.)

Real-time updates (optional Mercure)

The graph degrades gracefully by design:

Setup	Behaviour
`symfony/mercure-bundle` installed + hub configured	live push over Mercure (SSE)
running execution, no Mercure	polls `/state` and `/logs`
finished execution	fully static graph from persisted state

Nothing is required to get the static/poll graph. Install symfony/mercure-bundle to light up real-time — the bundle then auto-registers a Mercure publisher (MercureExecutionStatePublisher) and streams state + logs from the worker as the chain runs; otherwise a no-op publisher is used. Pass the hub's public URL + topic to the partial via a mercure: {url, topic} variable to enable the client side.

Real-time only applies to executions run asynchronously (a messenger worker); with the sync transport the chain runs in-request and the graph is static.

Adding your own chain operation

To add your own chain operation you need 2 classes. The operation itself that we will call MyVendor\Etl\Operation\OurTestOperation, and a MyVendor\Etl\OperationFactory\OurTestOperationFactory factory to create it. The factory allows us to configure the operation and inject service to our operation.

All operations needs to implement DataChainOperationInterface; they can extend AbstractChainOperation.

All factories needs to extend Oliverde8\Component\PhpEtl\Builder\Factories\AbstractFactory.

The operation is a Model and not a service, you therefore need to add the path to the exclusions so that it's not made a service by symfony:

App\:
  resource: '../src/'
  exclude:
    - '../src/Etl/Operation'

Factories needs to be tagged `etl.operation-factory\ . To remove the need to tag all your factories you can add the following line your your services.yaml file

    MyVendor\Etl\OperationFactory\:
        resource: '../src/Etl/OperationFactory/'
        tags: ['etl.operation-factory']

For more information on how the etl works and how to create operations check the Php Etl Documentation

oliverde8 / php-etl-bundle

Package info

Security

Statistics