ideative/data_handler_queue

Provides an API for registering DataHandler data and commands and replay them asynchronously at a later point in time.

0.0.5 2022-05-02 13:52 UTC

Requires

• ext-json: *
• typo3/cms-core: >=8.7.0,<10.0

GPL-2.0-or-later

• Francois Suter

README

This TYPO3 extension provides an API to register calls to the DataHandler. This can be data to be stored or TCE commands. The actual execution of the commands or storage of the data happens only when the command-line controller is invoked, emptying the queue of stored data and commands.

The need arose from having very intensive TCE operations to perform which eventually ran into timeouts or memory limits. This extension lets you choose how many entries you want to run each time.

Warning: this code should be considered to be still an alpha version. It was developed for a specific need and, although quite generic, it may fail you in your situation. In particular splitting TCE data and commands in lots may not be appropriate depending on what you are trying to do (for example, create a bunch of new elements in the TYPO3 database, related to one another, and expecting that all NEW*** constructs will be resolved properly).

Usage

Using the API

The API can be used in a single after instantiating the right class:

$queueUtility = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\Ideative\DataHandlerQueue\Utility\QueueUtility::class);
$queueUtility->store(
	'pages',
	42,
	'Zaphod Beeblebrox',
	'title'
); 

The above example stores the instruction to update the title of record 42 in the pages "pages" table to some value ("Zaphod Beeblebrox"). If the page were new, you would use the NEW*** syntax instead of a number.

For a command, the "field" argument is left empty and the fifth argument is used instead:

$queueUtility = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\Ideative\DataHandlerQueue\Utility\QueueUtility::class);
$queueUtility->store(
	'tx_deathstar_domain_model_planet',
	23,
	1,
	'',
	'delete'
); 

In this case, we indicate that planet 23 must be deleted. The "value" in this case depends on the TCE command used.

Executing the stored entries

A command-line tool is available to execute the stored TCE entries:

vendor/bin/typo3 datahandlerqueue:execute --limit=200

This will execute the first 200 entries (entries are ordered by primary key; we consider that the first entries to have been registered are the first that need to be executed). You can omit the "limit" argument, in which case the number of entries defaults to 100.

When entries are executed they are first marked as executed in the database (the "execute" field is set to 1), then passed through the DataHandler and only then deleted. This was designed so that crashes (due to timeouts or memory limits being reached) can be handled gracefully. Indeed if the script crashes while going through the DataHandler, you can see which entries were not executed and try to run them again. Of course, you need to backup and restore your database to use such a feature safely.

To make entries marked as executed available again, run the "reset" command:

vendor/bin/typo3 datahandlerqueue:reset

Credits

The extension icon is derived from a work by Kirby Wu from the Noun Project.