README

Writes Doctrine entities as multi-row INSERT, upsert and UPDATE statements, bypassing the ORM's UnitOfWork. Built for imports, syncs and other jobs that write thousands of rows, where persist() / flush() costs an identity map entry, change tracking and one statement per entity.

One statement per batch - rows travel as a single multi-row statement sized to the platform's bind parameter limit.
Plays by the mapping's rules - reads values through Doctrine types, resolves to-one associations to their foreign keys, and leaves out every column the database fills in by itself.
Three operations - plain insert, insert-or-overwrite (upsert) with a selectable set of overwritten fields, and partial update by field maps.
Test doubles included - in-memory factories that enforce the same field rules as the real writer, so service tests need no database.

Supports MySQL / MariaDB, PostgreSQL and SQLite on PHP >= 8.4 and doctrine/orm ^3.6.

Installation

composer require shredio/bulk-operations

Quick start

Everything goes through one interface, Shredio\BulkOperations\BulkOperationFactory. The real implementation wraps a Doctrine ManagerRegistry:

use Shredio\BulkOperations\Doctrine\DoctrineBulkOperationFactory;

$factory = new DoctrineBulkOperationFactory($managerRegistry);

$insert = $factory->createInsert(StockPrice::class);

foreach ($parsedRows as $row) {
	$insert->addEntity(StockPrice::fromCsvRow($row));
}

$written = $insert->flush(); // rows written in total

addEntity() buffers; whenever the buffer reaches the batch size, one multi-row INSERT is executed. flush() writes the remainder and returns the total number of rows sent. The batch size defaults to the platform's bind parameter limit divided by the column count and can be overridden per operation:

$insert = $factory->createInsert(StockPrice::class, batchSize: 500);

The examples below use this entity - daily stock data with an application-assigned composite key, which is the intended workload:

#[ORM\Entity]
#[ORM\Table(name: 'stock_price')]
class StockPrice
{

	#[ORM\Id]
	#[ORM\Column(length: 16)]
	public string $ticker;

	#[ORM\Id]
	#[ORM\Column(length: 10)]
	public string $tradedOn;

	#[ORM\Column]
	public int $volume;

	#[ORM\Column(type: 'datetime_immutable', updatable: false)]
	public DateTimeImmutable $firstSeenAt;

	#[ORM\Column(nullable: true)]
	public ?string $note;

}

Upsert: insert or overwrite

createUpsert() inserts rows and overwrites those that conflict on the entity's primary key - ON CONFLICT ... DO UPDATE on PostgreSQL and SQLite, ON DUPLICATE KEY UPDATE on MySQL:

$upsert = $factory->createUpsert(StockPrice::class);
$upsert->addEntity($todaysPrice); // inserted, or overwritten if the key exists
$upsert->flush();

By default a conflicting row has every field overwritten that the mapping allows an update to touch. That deliberately leaves out:

the key the rows conflicted on,
every column the database fills in (generated columns, generated identifiers, insertable: false),
fields mapped updatable: false - a re-import never moves firstSeenAt.

Choosing what an upsert overwrites

UpsertFields narrows the default from either end:

use Shredio\BulkOperations\UpsertFields;

// Overwrite only the named fields; everything else keeps its stored value.
$factory->createUpsert(StockPrice::class, UpsertFields::only(['volume', 'note']));

// Overwrite everything the mapping allows, except the named fields.
$factory->createUpsert(StockPrice::class, UpsertFields::except(['note']));

// Overwrite nothing: an insert that skips rows it already has.
$factory->createUpsert(StockPrice::class, UpsertFields::none());

Naming a field the mapping refuses is an error, not a silent no-op - only(['firstSeenAt']) throws InvalidFieldException and says why, so a misspelled field cannot quietly change nothing. except() tolerates naming a refused field, because excluding it only restates what the mapping already says.

UpsertFields::none() is the portable half of INSERT IGNORE: only a conflict on the key is swallowed. A NOT NULL violation or a bad foreign key still fails the batch, where MySQL's own INSERT IGNORE would downgrade them to warnings.

MySQL caveat: MySQL reacts to a conflict on any unique index, not just the primary key - the platform offers no way to choose the conflict target. On PostgreSQL and SQLite the conflict target is exactly the identifier columns.

Partial updates

createUpdate() rewrites chosen columns of existing rows. Rows are plain field maps, not entities - suited for corrections where hydrating entities would be wasteful:

// Matches on the identifier by default.
$update = $factory->createUpdate(StockPrice::class, ['volume', 'note']);
$update->addRow(['ticker' => 'AAPL', 'tradedOn' => '2026-08-03', 'volume' => 1_500, 'note' => 'corrected']);
$update->addRow(['ticker' => 'MSFT', 'tradedOn' => '2026-08-03', 'volume' => 2_250, 'note' => null]);
$update->flush();

One batch is one statement: the rows travel as a derived table joined against the target table, so a thousand corrections are one round trip.

Every row must carry exactly the operation's fields - the match fields plus the written fields, nothing more or less. A short row would bind values to the wrong columns, so it throws instead.
An update never inserts. A row that matches nothing is silently skipped. flush() returns rows sent, not rows changed - platforms disagree on whether rewriting a row with its own values counts as "affected", so an affected count would not mean the same thing twice.
Match fields default to the identifier and may be any columns, unique or not:

// Matching on a non-unique column updates every row it hits.
$update = $factory->createUpdate(StockPrice::class, ['isHalted'], matchFields: ['ticker']);
$update->addRow(['ticker' => 'AAPL', 'isHalted' => true]); // halts every AAPL row

Values are converted through their Doctrine type - custom types work, a BackedEnum may be passed as the case or its backing value, and a to-one association accepts either the related entity or the foreign key value itself. Where the target keys on a value object, that value object is the key value, and its own type converts it:

// All three name the same foreign key.
$update->addRow(['id' => 20, 'instrument' => $instrument]);
$update->addRow(['id' => 21, 'instrument' => new InstrumentId(2)]);
$update->addRow(['id' => 22, 'category' => 7]); // a target keyed on a plain int

Written fields are held to the mapping's update rules: updatable: false, generated columns and database-generated identifiers are refused for writing - while that same generated identifier is the most natural thing to match on.

Batches, transactions, limits

Default batch size fills the platform's bind parameter limit: 65 535 parameters on MySQL and PostgreSQL, 32 766 on SQLite, divided by the column count.

An update carries its rows as a derived table, and on most platforms that is a chain of UNION ALL the server parses recursively - a ceiling reached long before the bind limit. Its batches are therefore clamped, including a batch size you named; a smaller batch writes exactly the same rows.

Platform	Rows per update statement	Why
MySQL 8.0.19+	bind limit	rows travel in one `VALUES ROW(...)` list, so nothing recurses
MariaDB, MySQL 5.7	1 000	`thread_stack` (error 1436, `Thread stack overrun`)
PostgreSQL	2 000	`max_stack_depth` (`stack depth limit exceeded`)
SQLite	499	`SQLITE_MAX_COMPOUND_SELECT`

No write is wrapped in a transaction. Each batch commits on its own; a failure in batch three leaves batches one and two written. Callers that need all-or-nothing wrap the loop:

$connection->transactional(function () use ($factory, $entities): void {
	$insert = $factory->createInsert(StockPrice::class);

	foreach ($entities as $entity) {
		$insert->addEntity($entity);
	}

	$insert->flush();
});

What bypassing the UnitOfWork means

The ORM never sees these writes: no lifecycle events, no cascades, no identity map updates. Consequences worth knowing:

Entities already loaded in an EntityManager will not reflect the written values until re-read ($entityManager->clear() or a fresh query).
Nothing cascades. A to-one association is written as its foreign key columns; the related entity itself is not persisted.
To-many collections are refused, not ignored. An entity carrying unsaved items in a collection throws a LogicException rather than dropping them silently - write those items with their own bulk operation, or clear the collection first. Clean, lazily-loaded collections pass without triggering a load.

Symfony

The package ships a bundle (Symfony ^7.4 || ^8.0, needs DoctrineBundle for the ManagerRegistry):

// config/bundles.php
return [
	// ...
	Shredio\BulkOperations\Bridge\Symfony\BulkOperationsBundle::class => ['all' => true],
];

It has no configuration. BulkOperationFactory autowires to the Doctrine implementation:

final readonly class DailyPriceImport
{

	public function __construct(
		private BulkOperationFactory $bulkOperations,
	) {}

}

Testing your services

Two in-memory factories implement the same interface, so a service under test cannot tell the difference. Neither needs an entity manager or a database - they read the mapping attributes by reflection.

Testing\FakeBulkOperationFactory records everything, which is what most assertions want:

use Shredio\BulkOperations\Testing\FakeBulkOperationFactory;

$factory = new FakeBulkOperationFactory();
$import = new DailyPriceImport($factory);

$import->run($csv);

self::assertCount(500, $factory->getWrittenEntities(StockPrice::class));
self::assertSame(['volume', 'note'], $factory->getOperations()[0]->updateFields); // what the upsert overwrites

Per operation, getWrittenBatches() and getBufferedEntities() expose the split between written and buffered rows, which makes batch boundaries testable without a database. Update operations record alongside under getUpdateOperations() / getWrittenRows($entityClass).

Testing\ValidatingBulkOperationFactory validates and discards - for tests that only care that the service would not blow up in production, without holding entities in memory.

Both doubles enforce the writer's rules: an unknown field, a field an upsert may not overwrite, an update row carrying the wrong fields or a value of the wrong type all throw the same InvalidFieldException with the same message the Doctrine implementation raises. A dedicated parity test in this repository holds the two implementations to the same answers, so a test passing against the double keeps meaning something in production.

One limit: the doubles read attribute mapping only - entities mapped by XML, or fields inside embeddables, are invisible to them (they end up stricter than the writer, never looser).

Errors

All failures are subclasses of Shredio\BulkOperations\Exception\RuntimeException:

Exception	Raised when
`UnknownEntityException`	no entity manager manages the class, or the entity has no writable columns
`UnsupportedPlatformException`	the connection's platform is not MySQL/MariaDB, PostgreSQL or SQLite
`InvalidFieldException`	a named field is unknown, refused by the mapping, or a row does not fit the operation's fields
`InvalidEntityException`	an entity of the wrong class is added to an operation

Shredio\BulkOperations\Exception\LogicException (unchecked) signals a to-many collection with unsaved items. Database-level failures - constraint violations, connection loss - surface as Doctrine DBAL exceptions, untouched.

License

MIT

shredio / bulk-operations

Maintainers

Package info

Statistics

Security