fsc / batch
Library with classes to help you do batch.
Installs: 5 320
Dependents: 0
Suggesters: 0
Security: 0
Stars: 18
Watchers: 5
Forks: 3
Open Issues: 2
Requires
- php: >=5.3
- pagerfanta/pagerfanta: dev-master
- symfony/event-dispatcher: >=2.1
Requires (Dev)
- doctrine/common: >=2.2
- symfony/console: 2.*
This package is auto-updated.
Last update: 2024-11-12 04:24:16 UTC
README
PHP 5.3 library to help you run huge batch.
It takes a PagerfantaAdapterInterface (doctrine orm, propel, array, solarium etc... available) as a data source, will get data in slice of the size of the batch size, and will then process each context in batch by calling the callback you provided.
<?php use FSC\Batch\Batch; use FSC\Batch\Event\ExecuteEvent; $usersAdapter = ...; $solrIndex = ...; $batch = new Batch($usersAdapter, function (ExecuteEvent $event) use ($solrIndexer) { $solrIndexer->indexUser($event->getContext()); }); $batch->run(10); // Execute in batch of 10
Features:
- Evented system using the symfony2 event dispatcher, to easily be able to add behaviors
Included listeners:
- ProgressEventListener: Displays progress, elapsed time and estimated remaining time at the end of each batch.
- DoctrineEventListener: at the end of each batch:
- flush() the object manager, to save everything at the same time (may improve performance in some cases)
- clear() the object manager, to avoid memory leaks
Extra:
- Add a
PagerfantaAdapter
for doctrine ORM, that traverse the table using range queries on the id instead of LIMIT/OFFSET. LIMIT/OFFSET degrades query time as the OFFSET grows, wheareas range queries time stay consistent.
Be aware that this library is a WIP, and requires more tests.
Examples
Simple batch
<?php require_once __DIR__.'/../vendor/autoload.php'; use FSC\Batch\Batch; use FSC\Batch\Event\ExecuteEvent; use FSC\Batch\EventListener\ProgressEventListener; use Pagerfanta\Adapter\ArrayAdapter; use Symfony\Component\Console\Output\ConsoleOutput; $passwords = range(1, 100); $hashes = array(); $batch = new Batch(new ArrayAdapter($passwords), function (ExecuteEvent $event) use (&$hashes) { $hashes[] = crypt($event->getContext(), '$2a$10$'); }); $batch->getEventDispatcher()->addSubscriber(new ProgressEventListener(new ConsoleOutput())); $batch->run(10);
Would output
$ php examples/array_closure.php
Batch run start. 100 jobs [Mem: 0.52 MB]
[ 10/100] [ 10.00 %] ([Δ 0.83 sec] - [Elapsed 0.83 sec] - [Remaining 7 secs]) [Mem: 0.52 MB]
[ 20/100] [ 20.00 %] ([Δ 0.83 sec] - [Elapsed 1 sec] - [Remaining 6 secs]) [Mem: 0.52 MB]
[ 30/100] [ 30.00 %] ([Δ 0.83 sec] - [Elapsed 2 secs] - [Remaining 5 secs]) [Mem: 0.52 MB]
[ 40/100] [ 40.00 %] ([Δ 0.83 sec] - [Elapsed 3 secs] - [Remaining 4 secs]) [Mem: 0.52 MB]
[ 50/100] [ 50.00 %] ([Δ 0.82 sec] - [Elapsed 4 secs] - [Remaining 4 secs]) [Mem: 0.52 MB]
[ 60/100] [ 60.00 %] ([Δ 0.83 sec] - [Elapsed 4 secs] - [Remaining 3 secs]) [Mem: 0.52 MB]
[ 70/100] [ 70.00 %] ([Δ 0.83 sec] - [Elapsed 5 secs] - [Remaining 2 secs]) [Mem: 0.79 MB]
[ 80/100] [ 80.00 %] ([Δ 0.83 sec] - [Elapsed 6 secs] - [Remaining 1 sec]) [Mem: 0.79 MB]
[ 90/100] [ 90.00 %] ([Δ 0.83 sec] - [Elapsed 7 secs] - [Remaining 0.83 sec]) [Mem: 0.79 MB]
[100/100] [100.00 %] ([Δ 0.82 sec] - [Elapsed 8 secs] - [Remaining 0 sec]) [Mem: 0.79 MB]
Batch run end. took 0.82 sec [Mem: 0.79 MB]
Doctrine ORM Batch in a symfony command
This example uses the DoctrineEventListener, which flush (save everything) and clears the objectManager (avoid memory problems) at the end of each batch.
We also use a custom PagerfantaAdapter: DoctrineBatchAdapter
, that uses range queries (id > 100 AND id < 200) instead of LIMIT/OFFSET to avoid increasing query time as the OFFSET grows.
<?php use FSC\Batch\Batch; use FSC\Batch\Command\BatchCommand; use FSC\Batch\EventListener\DoctrineEventListener; use FSC\Batch\Event\ContextEvent; use Pagerfanta\Adapter\DoctrineORMAdapter; class UserIndexSolrCommand extends BatchCommand { protected function createBatch() { $em = $this->getContainer()->get('doctrine')->getEntityManager(); $qb = $em->getRepository('User')->createQueryBuilder('u'); $batch = new Batch(new DoctrineORMAdapter($qb), array($this, 'indexUser')); $batch->getEventDispatcher()->addSubscriber(new DoctrineEventListener($em)); return $batch; } public function indexUser(ContextEvent $event) { $user = $event->getContext(); // Index this user! } }