oat-sa / extension-tao-task-queue
TAO specific Task Queue with custom GUI
Installs: 56 909
Dependents: 2
Suggesters: 0
Security: 0
Stars: 2
Watchers: 46
Forks: 1
Open Issues: 3
Type:tao-extension
Requires
- ext-pdo: *
- evenement/evenement: ^2.0
- oat-sa/generis: >=15.22
- oat-sa/oatbox-extension-installer: ~1.1||dev-master
- oat-sa/tao-core: >=53.6.0||53.3.3
- react/child-process: ^0.5.2
- dev-master
- v6.9.0
- v6.8.6
- v6.8.5
- v6.8.4
- v6.8.3
- v6.8.2
- v6.8.1
- v6.8.0
- v6.7.3
- v6.7.2
- v6.7.1
- v6.7.0
- v6.6.5
- v6.6.4
- v6.6.3
- v6.6.2
- v6.6.1
- v6.6.0
- v6.5.3
- v6.5.2
- v6.5.1
- v6.5.0
- v6.4.0
- v6.3.1
- v6.3.0
- v6.2.0
- v6.1.2
- v6.1.1
- v6.1.0
- v6.0.0
- v5.5.1
- v5.5.0
- v5.4.1.1
- v5.4.1
- v5.4.0
- v5.3.1
- v5.2.0
- v5.1.0
- v5.0.1
- v5.0.0
- v4.0.0
- v3.3.1.1
- v3.3.1
- v3.3.0
- v3.1.0
- v2.2.0
- v2.1.0
- v2.0.0
- v1.4.0
- v1.3.0
- v1.2.1
- v1.2.0
- v1.1.0
- v0.17.4
- v0.17.3
- v0.17.1
- v0.17.0
- v0.16.0
- v0.15.0
- v0.14.3
- v0.14.2
- v0.13.1
- v0.11.0
- v0.7.0
- v0.5.1
- v0.5.0
- v0.4.2
- v0.4.1
- v0.4.0
- v0.2.2
- v0.2.1
- v0.2.0
- v0.1.1
- v0.1.0
- dev-feat/SI-610/cleaning-script
- dev-l10n_develop
- dev-develop
- dev-fix/REL-1580/crowdin-translations-compiled
- dev-fix/PISA25-674/fix-count-triggers-error-in-case-of-null
- dev-feature/sonarqube-integration
- dev-breaking/TDR-9/handlebars-4-upgrade
- dev-backport/adf-1571/6.8.0.1
- dev-fix/update-autoRelease-action-with-bot-user
- dev-release-6.6.5
- dev-backport/fix/AUT-2995/fix_br_target_language
- dev-fix/AUT-2995/fix_br_target_language
- dev-release-6.6.4
- dev-hotfix/AUT-2995/add-brazilian-portuguese-language
- dev-backport/REL-470_translations
- dev-fix/translation_file_integrity
- dev-release-2021-08-lts
- dev-feature/ADF-305/separate_tasks_for_indx
- dev-feature/CGF-205/taskQueueLogging
- dev-release/v3.3.1.1
- dev-feature/NEX-1135/task_association_cli_control
- dev-chore/TECEN-118-add-composer-lock-and-vendor-to-gitignore
- dev-fix/NEX-517/spanner_support
- dev-fix/NCC-152/AssignmentsReportKeepsTracksOfAssignment
- dev-feature/update-dependencies-and-doc
- dev-feature/NEX-519/add-transactional
- dev-feature/TAO-7484/hide-actions-with-annotations
- dev-read-replicas-in-task-queue-prototype
- dev-feature/TAO-7083/task-queue-manager
- dev-feature/TAO-5746/url-building-for-resource
- dev-fix/lazyload-queue
- dev-feature/TAO-4578/grab-file-from-different-storages
This package is auto-updated.
Last update: 2024-11-14 10:02:34 UTC
README
This article describes the functioning of an ASYNC Task Queue.
Install
You can add the Task Queue as a standard TAO extension to your current TAO instance.
$ composer require oat-sa/extension-tao-task-queue
Queue component
Queue can work with different types of queue brokers, here two types are to accomplish ASYNC mechanism:
- RdsQueueBroker which stores tasks in RDS.
- SqsQueueBroker which is for using AWS SQS.
Note:
When SqsQueueBroker is used, please make sure that "oat-sa/lib-generis-aws" is included in the main composer.json and you have generis/awsClient.conf.php properly configured.
Weight
A Queue can have a weight. If multiple Queues are in use, this weight will be used for randomly select a Queue to be consumed. For example, if QueueA has weight of 1 and QueueB has weight of 2, then QueueB has about a 66% chance of being selected.
Worker component
Here we have a so called LongRunningWorker
which can run unlimited time.
It has built-in signal handling for the following actions:
- Shutting down the worker gracefully: SIGTERM/SIGINT/SIGQUIT
- Pausing task processing: SIGUSR2
- Resuming task processing: SIGCONT
Note:
Multiple workers can be run at the same time.
After processing the given task, the worker saves the generated report for the task through the Task Log.
Service setup examples
Multiple Queues settings
In this case we have 3 Queues registered: one of them is using SQS broker, the other two RDS. Every Queue has its own weight (like 90, 30, 10) which will be used at selecting the next queue to be consumed.
And we have two tasks linked to different queues, furthermore the default queue is specified ('background') what will be used for every other tasks not defined in OPTION_TASK_TO_QUEUE_ASSOCIATIONS.
use oat\tao\model\taskQueue\QueueDispatcher; use oat\tao\model\taskQueue\Queue; use oat\taoTaskQueue\model\QueueBroker\RdsQueueBroker; use oat\taoTaskQueue\model\QueueBroker\SqsQueueBroker; use oat\tao\model\taskQueue\TaskLogInterface; use oat\tao\model\taskQueue\QueueDispatcherInterface; $queueService = new QueueDispatcher(array( QueueDispatcherInterface::OPTION_QUEUES => [ new Queue('priority', new SqsQueueBroker('default', \common_cache_Cache::SERVICE_ID, 10), 90), new Queue('standard', new RdsQueueBroker('default', 5), 30), new Queue('background', new RdsQueueBroker('default', 5), 10) ], QueueDispatcherInterface::OPTION_TASK_LOG => TaskLogInterface::SERVICE_ID, QueueDispatcherInterface::OPTION_TASK_TO_QUEUE_ASSOCIATIONS => [ SomeImportantAction::class => 'priority', SomeLessImportantTask::class => 'standard' ] )); $queueService->setOption(QueueDispatcherInterface::OPTION_DEFAULT_QUEUE, 'background'); $this->getServiceManager()->register(QueueDispatcherInterface::SERVICE_ID, $queueService);
If the queue has not been initialized, meaning the required queue container has not been created yet:
try { $queueService->initialize(); } catch (\Exception $e) { return \oat\oatbox\reporting\Report::createError('Initializing queues failed'); }
Initializing the queues and the task log container
You can run this script if you want to be sure that the required queues and the task log container are created.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue'
Note:
This script also can be used to change the current queues to use a different queue broker.
- Changing every existing queue to use InMemoryQueueBroker. (Sync Queue)
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue' --broker=memory
- Changing every existing queue to use RdsQueueBroker. Option "persistence" is required, "receive" (Maximum amount of tasks that can be received when polling the queue) is optional.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue' --broker=rds --persistence=default --receive=10
- Changing every existing queue to use SqsQueueBroker. Option "aws-profile" is required, "receive" is optional.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue' --broker=sqs --aws-profile=default --receive=10
- If you want to apply the settings above for a specific queue, you can add
--queue=...
option to the command. In the following case, onlymyQueue
will be modified.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue' --queue=myQueue --broker=rds --persistence=default --receive=10
- Setting a task selector strategy.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\InitializeQueue' --strategy="\oat\taoTaskQueue\model\TaskSelector\StrictPriorityStrategy"
Running a worker
To run a worker, use the following command. It will start a worker for running infinitely and iterating over every registered Queues based in their weights.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\RunWorker'
If you want the worker running for a dedicated Queue, pass the name of the queue to the command like this:
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\RunWorker' --queue=priority
You can limit the iteration of the worker. It can be used only on a dedicated queue.
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\RunWorker' --queue=standard --limit=5
If you want to associate specyfic task to new queue you can use this command:
$ sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\ManageAssociationMap' \ -t '{ you fully qualified task class name }' -q queue-name
Next time when defined task will be created, it will be assign to specified queue.
Summarize stuck tasks
Execute this command if you want to summarize stuck tasks. Example:
sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\StuckTaskSummary' \ --queue indexation_queue \ --age 300 \ --whitelist "oat\tao\model\search\tasks\UpdateResourceInIndex,oat\tao\model\search\tasks\UpdateClassInIndex"
Restart stuck tasks
Execute this command if you want to restart stuck tasks. Example:
sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\StuckTaskRestart' \ --queue indexation_queue \ --age 300 \ --whitelist "oat\tao\model\search\tasks\UpdateResourceInIndex,oat\tao\model\search\tasks\UpdateClassInIndex"
Rest API
The task log reports can be viewed/consume using the Application Programming Interface (API). In order to use it please check the swagger file in (doc/taskApi.yml).
Command Line Utility
Besides using the API to check reports of tasks, another way it's using the command line.
sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --help
This command will show you all the possibilities action the the utility can have.
Examples
1. Stats
Description: Return stats about the tasks logs statuses
Example: sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --stats
2. List Task Logs
Description: List All the tasks that are not archived will be retrived, default limit is 20
Example: sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --available --limit[optional]=20 --offset[optional]=10
3. Get Task Log
Description: Get an specific task log by id
Example: sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --get-task=[taskdId]
4. Archive a Task Log
Description: Archive a task log
Example: sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --archive=[taskdId] --force[optional]
5. Cancel a Task Log
Description: Cancel a task log
Example: sudo -u www-data php index.php 'oat\taoTaskQueue\scripts\tools\TaskLogUtility' --cancel=[taskdId] --force[optional]