atymic / laravel-bulk-sqs-queue
Laravel SQS Bulk Queue
Fund package maintenance!
atymic
Installs: 65 329
Dependents: 0
Suggesters: 0
Security: 0
Stars: 12
Watchers: 2
Forks: 3
Open Issues: 0
Requires
- php: ^8.1
- aws/aws-sdk-php: ^3
- illuminate/contracts: ^9.0 || ^10.0 || ^11.0
Requires (Dev)
- brianium/paratest: ^6.2
- nunomaduro/collision: ^7.0
- orchestra/testbench: ^7.0 || ^8.0 || ^9.0
- phpunit/phpunit: ^9.3
This package is auto-updated.
Last update: 2024-10-20 01:40:50 UTC
README
Installation
You can install the package via composer:
composer require atymic/laravel-bulk-sqs-queue
How it works
By default, Laravel allows you to easily execute a batch of jobs with Queue::bulk()
method or with built-in job batching. Both methods accept an array of jobs for dispatch and loop over every job, making one HTTP request for each job.
This isn't an issue when you are dispatching a few jobs, but there's two major issues with bigger batches:
- Waiting for 1000 http requests (even SQS, with 20-50ms latency) is two to five seconds, that's slow!
- SQS is billed per request, and they support batching of up to 10 messages for the same cost as a single
sendMessage
call. That's a 10x cost saving!
But AWS SQS has a batch action that let you group up to 10 messages with a single request, in order to reduce costs. Under the hood, this package override the bulk method to:
- Batch jobs into 10 per request, or 200kb chunks (SQS has a maximum of 256kb, including request overhead/etc)
- Dispatch those batches asynchronously, up to
$concurrency
at a time (default 5)
That's about it. With this package, the laravel queue system should work the exact same as normal. You should have the same result in your application and your AWS SQS dashboard but with a smaller AWS bill :)
Usage
This package provides a queue connector called sqs-bulk
. Inside your queue.php
config file, add it to connections
:
'connections' => [ 'sqs-bulk' => [ 'driver' => 'sqs-bulk', 'key' => env('AWS_ACCESS_KEY_ID'), 'secret' => env('AWS_SECRET_ACCESS_KEY'), 'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'), 'queue' => env('SQS_QUEUE', 'default'), 'suffix' => env('SQS_SUFFIX'), 'region' => env('AWS_DEFAULT_REGION', 'us-east-1'), 'after_commit' => false, 'concurrency' => 5, // Set the request concurrency, defaults to 5 ], ]
Then you can start a queue worker for the new "connection" and the given queue ('default' queue can be override with SQS_QUEUE
):
php artisan queue:work sqs-bulk --queue=default
It will process new jobs as they are pushed onto the queue. You can group jobs with queue's bulk method:
Illuminate\Support\Facades\Queue::bulk([ new \App\Jobs\Foo, new \App\Jobs\Bar, new \App\Jobs\Baz, ], '', 'default');
or with laravel's built-in job batching feature:
Illuminate\Support\Facades\Bus::batch([ new \App\Jobs\Foo, new \App\Jobs\Bar, new \App\Jobs\Baz, ])->name('My sqs batch')->onQueue('default')->dispatch();
It should have processed the 3 jobs by creating 3 "messages" on AWS SQS but with only 1 request.
Failing jobs
This package only affect the way jobs are transmitted to AWS SQS. Once received by AWS SQS, they are handled as separate messages. From AWS SQS perspective, there is no relation between messages. The batch request can result in a combination of successful and unsuccessful actions that doesn't affect each others.
Testing
composer test
Changelog
Please see CHANGELOG for more information on what has changed recently.
Contributing
Please see CONTRIBUTING for details.
Security Vulnerabilities
Please review our security policy on how to report security vulnerabilities.
Credits
License
The MIT License (MIT). Please see License File for more information.