wwwision/import-service

Generic service for importing data from different sources to configurable targets such as the Neos Content Repository or an arbitrary database table

3.0.1 2024-06-27 09:56 UTC

Requires

• php: ^8.1
• ext-json: *
• guzzlehttp/guzzle: ^7.0
• neos/flow: ^6.0 || ^7.0 || ^8.0

Requires (Dev)

• roave/security-advisories: dev-latest

MIT

README

Neos Flow package for importing data from different sources to configurable targets such as the Neos Content Repository or an arbitrary database table

Usage

Setup

Install this package using composer via

composer require wwwision/import-service

Define an Import Preset

Add some Import Preset configuration to your projects Settings.yaml, for example:

Wwwision:
  ImportService:
    presets:

      'some-prefix:some-name':
        source:
          factory: 'Wwwision\ImportService\DataSource\Http\HttpSourceFactory'
          options:
            endpoint: 'https://some-endpoint.tld/data.json'
        target:
          factory: 'Wwwision\ImportService\DataTarget\Dbal\DbalSourceFactory'
          options:
            table: 'some_table'
        mapping:
          'id': 'id'
          'given_name': 'firstName'
          'family_name': 'lastName'

Run the import

./flow import:run some-prefix:some-name

Pre-process data

Sometimes the data has to be processed before it is mapped to. This can be done with a dataProcessor.

Example:

Implementation

A processor is any public method of any class that can be instantiated by Flow without additional arguments:

<?php
declare(strict_types=1);
namespace Some\Package;

use Wwwision\ImportService\ValueObject\DataRecord;
use Wwwision\ImportService\ValueObject\DataRecords;

final class SomeProcessor
{

    public function someMethod(DataRecords $records): DataRecords
    {
        return $records->map(static fn (DataRecord $record) => $record->withAttribute('title', 'overridden'));
    }
}

Note: The processor class can have dependencies, but it should be possible to create a new instance via ObjectMananger::get($processorClassname) without further arguments, i.e. the class should behave like a singleton (see https://flowframework.readthedocs.io/en/stable/TheDefinitiveGuide/PartIII/ObjectManagement.html)

Configuration

Wwwision:
  ImportService:
    presets:
      '<preset-name>':
        # ...
        options:
          dataProcessor: 'Some\Package\SomeProcessor::someMethod'

Note: The syntax looks like the method has to be static, but that's not the case. It just has to satisfy PHPs is_callable() function

Validate configuration

Configuration for this package is verbose and thus error-prone. The settings can be validated against a schema via the following command:

./flow configuration:validate --type Settings --path Wwwision.ImportService

Which should produce the output

Validating Settings configuration on path Wwwision.ImportService

All Valid!

CLI

This package provides the following CLI commands:

import:run

Synchronizes data (add, update,delete) for the given preset.

Usage

./flow import:run [<options>] <preset>

Arguments

  --preset             The preset to use (see Wwwision.Import.presets setting)

Options

  --quiet                   If set, no output, apart from errors, will be displayed
  --force-updates           If set, all local records will be updated regardless of their version/timestamp. This is useful for node type changes that require new data to be fetched
  --from-fixture            If set, the data will be loaded from a local fixture file instead of the configured data source
  --override-source-options Allows to override default options for the data source via JSON, e.g. '{"endpoint":"https://some-custom.tld/endpoint"}'
  --override-target-options Allows to override default options for the data target via JSON, e.g. '{"endpoint":"https://some-custom.tld/endpoint"}'

import:prune

Removes all data for the given preset

Usage

./flow import:prune [<options>] <preset>

Arguments

  --preset             The preset to reset (see Wwwision.Import.presets setting)

Options

  --assume-yes         If set, "yes" will be assumed for the confirmation question

import:presets

Lists all configured preset names

Usage

./flow import:presets

import:preset

Displays configuration for a given preset

Usage

./flow import:preset <preset>

Arguments

  --preset             The preset to show configuration for (see Wwwision.Import.presets setting)

import:setup

Set up the configured data source and target for the specified preset and/or display status

Usage

./flow import:setup <preset>

Arguments

  --preset             Name of the preset to set up

Neos ContentRepository import

Importing data from a 3rd party API into the Neos Content Repository is one of the main requirements for this package.

To import records into the Content Repository, the privided ContentRepositoryTarget can be used, it has the following options:

nodeType (string, required): Type of the imported nodes
rootNodePath: (string, optional): Absolute path of the root node underneath imported nodes will placed
parentNodeResolver: (callable string, optional): Callback in the format ('<FQCN>::<methodName>') that is invoked to resolve the parent node for a given record. The signature is: \Closure(\Wwwision\ImportService\ValueObject\DataRecord): \Neos\ContentRepository\Domain\Model\Node
nodeVariantsResolver: (callable string, optional): Callback in the format ('<FQCN>::<methodName>') that is invoked to resolve dimension variants for a given record. The signature is: \Closure(\Wwwision\ImportService\ValueObject\DataRecord): array of dimension values
idPrefix: (string, optional): Optional prefix that is prepended to the dataRecord IDs in order to get globally unique node identifiers
softDelete (boolean, optional): If set, removed records lead to the corresponding node to be _disabled_ only. Otherwise they are deleted from the Content Repository
rootNodeType: (string, optional): Type of the root node, imported nodes are placed in – if set and no root node can be found, the importer tries to create it via ./flow importer:setup

Note: Either rootNodePath or parentNodeResolver have to be specified!

Example configuration:

Wwwision:
  ImportService:
    presets:

      'some:preset':
        source:
          # depending on the data source, see above
        target:
          factory: 'Wwwision\ImportService\DataTarget\ContentRepository\ContentRepositoryTargetFactory'
          options:
            nodeType: 'Some.Package:Type.Of.Imported.Nodes'
            rootNodePath: '/sites/some-site/some/path'
            # alternatively:
            # parentNodeResolver: Some`\Package\SomeSingleton::someMethod
            idPrefix: 'product-'
            softDelete: true
        mapping:
          'title': 'title' # just use the record title 1:1
          'price': '${record.priceNet + record.vat}' # arbitrary Eel expressions are supported
          'uriPathSegment': '${Some.Custom.Eelhelper(record.title + "-" + record.id)}' # ...including custom Eel helpers (registered via Neos.Fusion.defaultContext setting)

Usage without Neos.ContentRepository package

If the Neos.ContentRepository package is not installed Flow's proxy class builder throws an UnknownObjectException. Disable auto-wiring in Objects.yaml:

Wwwision\ImportService\DataTarget\ContentRepositoryTarget:
autowiring: false