ttree / contentrepositoryimporter
Helper package to import data in the Neos content repository
Installs: 9 764
Dependents: 0
Suggesters: 0
Security: 0
Stars: 15
Watchers: 6
Forks: 14
Open Issues: 1
Type:neos-package
Requires
- cocur/slugify: ^2.5
- ezyang/htmlpurifier: ^4.10.0
- neos/neos: ^7.0 || ^8.0
README
This package contains generic utility to help importing data in the Neos Content Repository.
What's included ?
- A command controller (CLI) to launch your import presets
- Based on simple conventions
- DataProvider: used to prepare and cleanup data from the external source
- Importer: get the data from the DataProvider and push everything in the CR
- DataType: Simple object used to cleanup value and share code between DataProvider
- Split your import in multiple sub commands to avoid high memory usage
- No big magic, you can always take control by overriding the default configuration and methods
A basic DataProvider
Every data provider must extend the DataProvider
abstract class or implement the
interface DataProviderInterface
. Check the source code of the abstract data provider, there are some useful things
to discover.
It's important to update the count
property when you process data from the external source. During the processing,
you can decide to skip some data (invalid data, missing values, ...) so we can not use the SQL count feature.
Try to do most of the data cleaning up in the data provider, so the data would arrive to the importer ready for insertion. Basically the array build by the provider should contains the data with the property name that match your node type property name. If you need to transport value that will not match the node properties, please prefix them with '_'.
There is some magic value, those values MUST be on the first level of the array:
- __identifier (optional) This UUID will be used in the imported node, you should use
AbstractImporter::applyProperties
to have this feature, used by default - __externalIdentifier (required) The external identifier of the data, this one is really important. The package keep track of imported data
- __label (required) The label of this record used by the importer mainly for logging (this value is not imported, but useful to follow the process) if you run twice the same import, the imported node will be updated and not created.
Tips: If the properties of your nodes are not at the first level of the array, you can override the method AbstractImporter::getPropertiesFromDataProviderPayload
Output of the provider
Your provider should output something like this:
[
'__label' => 'The external content lable, for internal use'
'__externalIdentifier' => 'The external external identifier, for internal use'
'title' => 'My title'
'year' => 1999
'text' => '...'
]
Tips: If your provider does not return an array, you MUST registrer a TypeConverter to convert it to an array. The property mapper is used automatically by the Importer.
Content Dimensions support
If your data provider follow this convention, the importer can automatically create variants of your nodes:
[
'__label' => 'The external content lable, for internal use'
'__externalIdentifier' => 'The external external identifier, for internal use'
'title' => 'My title'
'year' => 1999
'text' => '...',
'@dimensions' => [
'@en' => [
'@strategy' => 'merge',
'title' => '...',
],
'@fr' => [
'@strategy' => 'merge',
'title' => '...',
],
]
]
The @en
is a preset name, you must configuration the presets on your Settings.yaml
:
Ttree:
ContentRepositoryImporter:
dimensionsImporter:
presets:
fr:
language: ['fr', 'en', 'de']
en:
language: ['en', 'de']
de:
language: ['de']
Share data between preset parts
You can split your import in multiple parts. Each parts is executed in a separate request. Sometimes it's useful to share data between parts (ex. in the first
part you import the taxonomy, and in the second parts you map documents with the taxonomy). Those solve this use case, we integrate a feature called Vault. The
Vault is simply a cache accessible in the importer and data provider by calling $this->vault->set($key, $name)
and $this->vault->get($key)
. The
current preset is the namespace, so you can use simple keys like name, ids, ...
The cache is flushed if you call flow import:init --preset your-preset
.
Basic provider
class BasicDataProvider extends DataProvider { /** * @return array */ public function fetch() { $result = []; $query = $this->createQuery() ->select('*') ->from('demo_table', 'd') ->orderBy('d.name'); $statement = $query->execute(); while ($demoRecord = $statement->fetch()) { $result[] = [ '__externalIdentifier' => (integer)$demoRecord['id'], 'name' => String::create($demoRecord['name'])->getValue() ]; } $this->count = count($result); return $result; } }
A basic Importer
Every data importer must extend the AbstractImporter
abstract class or implement the interface ImporterInterface
.
In the processRecord
method you handle the processing of every record, such as creating Content Repository node for each incoming data record.
Do not forget to register the processed nodes with registerNodeProcessing
. The method will handle feature like logging and tracking of imported node to decide if the local node need to be created or updated.
class ProductImporter extends AbstractImporter { /** * @var string */ protected $externalIdentifierDataKey = 'productNumber'; /** * @var string */ protected $labelDataKey = 'properties.name'; /** * @var string */ protected $nodeNamePrefix = 'product-'; /** * @var string */ protected $nodeTypeName = 'Acme.Demo:Product'; /** * Starts batch processing all commands * * @return void * @api */ public function process() { $this->initializeStorageNode('shop/products', 'products', 'Products', 'products'); $this->initializeNodeTemplates(); $nodeTemplate = new NodeTemplate(); $this->processBatch($nodeTemplate); } }
A basic preset
You can configure an import preset in your Settings.yaml
. A preset is split in multiple parts. If you use the
batchSize
, the current part will be executed by batch, by using a sub CLI request. This can solve memory or
performance issue for big imports.
Ttree: ContentRepositoryImporter: sources: default: host: localhost driver: pdo_mysql dbname: database user: user password: password extraSourceDatabase: host: localhost driver: pdo_mysql dbname: database user: user password: password presets: 'base': parts: 'news': label: 'News Import' dataProviderClassName: 'Your\Package\Importer\DataProvider\NewsDataProvider' importerClassName: 'Your\Package\Importer\Importer\NewsImporter' 'page': label: 'Page Import' dataProviderClassName: 'Your\Package\Importer\DataProvider\PageDataProvider' dataProviderOptions: source: 'extraSourceDatabase' someOption: 'Some option that will be available in the options property of the data provider' importerClassName: 'Your\Package\Importer\Importer\PageImporter' importerOptions: siteNodePath: '/sites/my-site' someOption: 'Some option that will be available in the options property of the importer' batchSize': 120 'pageContent': label: 'Page Content Import' dataProviderClassName: 'Your\Package\Importer\DataProvider\PageContentDataProvider' importerClassName: 'Your\Package\Importer\Importer\PageContentImporter' batchSize: 120
Start your import process
Tips: Do not forget to require this package from the package in which you do the importing, to ensure the correct loading order, so the settings would get overriden correctly.
From the CLI:
flow import:batch --preset base
You can also filter the preset steps:
flow import:batch --preset base --parts page,pageContent
For testing purposes, or if you would like to override the value defined in your preset, you can also specify the number of records which should be imported at a time in an isolated sub-process:
flow import:batch --preset base --batch-size 50
Passing exceeding arguments to the DataProvider
The import process supports passing unnamed exceeding arguments to the DataProvider
. This can be useful if you e.g. want to
allow importing only a single record
flow import:batch --preset base recordIdentifier:1234
Exceeding arguments will be available in the DataProvider
through $this->getExceedingArguments()
. You need to process
this data yourself and apply it to your fetching logic.
Command based importers
Some data sources may consist of commands rather than data records. For example, a JSON file may contain create
,
update
and delete
instructions which reduce the guess-work on the importer's side, which records may be new,
which should be updated and if the absence of a record means that the corresponding node should be deleted from
the content repository.
For these cases you can extend the AbstractCommandBasedImporter
. If your data records contain a mode
field, the
importer will try to call a corresponding command method within the same class.
Consider the following data source file as an example:
[ { "mode": "create", "mpn": "1081251137", "languageIdentifier": "de", "properties": { "label": "Coffee Machine", "price": "220000", "externalKey": "1081251137" } }, { "mode": "delete", "mpn": "591500202" } ]
A corresponding ProductImporter
might look like this:
/** * Class ProductImporter */ class ProductImporter extends AbstractCommandBasedImporter { /** * @var string */ protected $storageNodeNodePath = 'products'; /** * @var string */ protected $storageNodeTitle = 'Products'; /** * @var string */ protected $externalIdentifierDataKey = 'mpn'; /** * @var string */ protected $labelDataKey = 'properties.Label'; /** * @var string */ protected $nodeNamePrefix = 'product-'; /** * @var string */ protected $nodeTypeName = 'Acme.MyShop:Product'; /** * Creates a new product * * @param string $externalIdentifier * @param array $data * @return void */ protected function createCommand($externalIdentifier, array $data) { $this->applyProperties($data['properties'], $this->nodeTemplate); $node = $this->storageNode->createNodeFromTemplate($this->nodeTemplate); $this->registerNodeProcessing($node, $externalIdentifier); } /** * Updates a product * * @param string $externalIdentifier * @param array $data * @return void */ protected function updateCommand($externalIdentifier, array $data) { $this->applyProperties($data['properties'], $this->nodeTemplate); $node = $this->storageNode->createNodeFromTemplate($this->nodeTemplate); $this->registerNodeProcessing($node, $externalIdentifier); } /** * Deletes a product * * @param string $externalIdentifier * @param array $data */ protected function deleteCommand($externalIdentifier, array $data) { // delete the product node } }
CSV Data Provider
This package comes with a basic data provider for CSV files which will suffice for many scenarios. The class name for
this data provider is Ttree\ContentRepositoryImporter\DataProvider\CsvDataProvider
.
The following options can be passed to the data provider:
csvFilePath
: the full path and filename leading to the file to importcsvDelimiter
: the delimiter used in the CSV file (default:,
)csvEnclosure
: the character which is used for enclosing the values (default:"
)skipHeader
: if the first line in the CSV file should be ignored (default: false)
Here is an example for a preset using the CSV Data Provider:
Ttree: ContentRepositoryImporter: presets: 'products': parts: 'products': label: 'Product Import' batchSize: 100 dataProviderClassName: 'Ttree\ContentRepositoryImporter\DataProvider\CsvDataProvider' dataProviderOptions: csvFilePath: '/tmp/Products.csv' csvDelimiter: ';' csvEnclosure: '"' skipHeader: true importerClassName: 'Acme\MyProductImporter\Service\Import\ProductImporter' importerOptions: siteNodePath: '/sites/wwwacmecom'
Acknowledgments
Development sponsored by ttree ltd - neos solution provider.
We try our best to craft this package with a lots of love, we are open to sponsoring, support request, ... just contact us.
License
Licensed under GPLv3+, see LICENSE