README

A CSV import engine for Laravel Eloquent models.

Installation

You can install the package via Composer:

composer require yossuf/laravel-import

You may publish all of the package's resources at once:

php artisan vendor:publish --tag="laravel-import"

Or, you may publish each resource individually:

Publishing the Configuration File

php artisan vendor:publish --tag="laravel-import-config"

The published config/import.php covers the tracking table, the queue and the default sizes, each with an environment variable:

return [
    // Where tracking records live and how long they are kept. A null
    // connection uses the application's default; a null prune_after
    // keeps every record.
    'tracking' => [
        'database' => env('IMPORT_CONNECTION'),
        'table' => env('IMPORT_TABLE', 'IMPORT'),
        'prune_after' => env('IMPORT_PRUNE_AFTER'),
    ],

    // Imports run as queued jobs; the "sync" connection runs them inline.
    // The lock values tune how the import jobs of one import wait for
    // each other; expire_after has to exceed the time a single one takes.
    'queue' => [
        'connection' => env('IMPORT_QUEUE_CONNECTION'),
        'queue' => env('IMPORT_QUEUE'),
        'release_after' => (int) env('IMPORT_QUEUE_RELEASE_AFTER', 5),
        'expire_after' => (int) env('IMPORT_QUEUE_EXPIRE_AFTER', 300),
    ],

    // Rows per queued job for ShouldChunkRead importers whose chunkSize()
    // returns null.
    'chunk_size' => (int) env('IMPORT_CHUNK_SIZE', 500),

    // Rows per upsert statement for ShouldBulkWrite importers whose
    // bulkSize() returns null.
    'bulk_size' => (int) env('IMPORT_BULK_SIZE', 500),
];

IMPORT_CONNECTION=mysql
IMPORT_TABLE=IMPORT
IMPORT_QUEUE_CONNECTION=redis
IMPORT_QUEUE=imports
IMPORT_QUEUE_RELEASE_AFTER=5
IMPORT_QUEUE_EXPIRE_AFTER=300
IMPORT_CHUNK_SIZE=500
IMPORT_BULK_SIZE=500

Running the Migrations

Publish the migrations into your application and run them:

php artisan vendor:publish --tag="laravel-import-migrations"
php artisan migrate

The table name and connection come from config/import.php, through IMPORT_TABLE and IMPORT_CONNECTION. The migrations read both from the model and hard-code neither.

Pruning Old Tracking Records

Tracking records are kept forever by default. Set prune_after to a relative period Carbon understands to let Laravel's model:prune command delete the older ones:

IMPORT_PRUNE_AFTER="30 days"

"6 months" and "1 year" work just as well. A record is aged by its created_at and pruned whatever state it ended in, so an import abandoned in Waiting or Processing is cleaned up too.

An invalid period fails the prune run with an InvalidArgumentException naming the value. That covers a typo such as "30" or "banana months", and a period that is not in the past, such as "-30 days" or "0 days", which would prune every record. Leaving prune_after unset is the only way to switch pruning off. Confirm a new period with --pretend before scheduling it.

The command only auto-discovers models in your own app/Models, so name this one explicitly when scheduling it:

use Illuminate\Support\Facades\Schedule;
use Yossuf\Laravel\Import\Models\ImportRecord;

Schedule::command('model:prune', [
    '--model' => [ImportRecord::class],
])->daily();

Check what would be deleted without deleting it:

php artisan model:prune --model="Yossuf\Laravel\Import\Models\ImportRecord" --pretend

Usage

Once the IMPORT table is migrated:

1. Write an importer for your model

Implement Yossuf\Laravel\Import\Contracts\ModelImport to say which Eloquent model a CSV file is imported into and how each row maps onto it. The package reads the file for you.

namespace App\Imports;

use App\Models\Product;
use Yossuf\Laravel\Import\Concerns\Importable;
use Yossuf\Laravel\Import\Contracts\ModelImport;

class ProductImport implements ModelImport
{
    use Importable;

    public function model(): string
    {
        return Product::class;
    }

    public function rules(): array
    {
        return [
            'sku' => ['required', 'string', 'max:255'],
            'name' => ['required', 'string', 'max:255'],
        ];
    }

    public function map(array $row): array
    {
        return [
            'sku' => $row['sku'],
            'name' => $row['name'],
        ];
    }

    public function uniqueBy(): array
    {
        return ['sku'];
    }
}

Rows reach rules() with the file's headers as snake_case keys, so a Product Name column is product_name. Every value is a trimmed string, an empty cell is null, and a value starting with = is kept as text rather than treated as a formula. A blank cell therefore fails required and passes nullable. A number arrives as the string the file holds, so cast it in map() when the column needs it.

Importable supplies engineOptions(). Override it when the file calls for it:

public function engineOptions(): array
{
    return ['delimiter' => ';', 'enclosure' => "'", 'encoding' => 'ISO-8859-1'];
}

An importer like the one above is read row by row, each row imported as it is read. To read a large file in chunks, each imported by its own queued job, implement ShouldChunkRead as well. Importable already supplies chunkSize(), which reads a $chunkSize property the way a queued job's $tries is read. Without the property, or with it null, the size is config('import.chunk_size'). Declare the property to pick a size:

use Yossuf\Laravel\Import\Contracts\ModelImport;
use Yossuf\Laravel\Import\Contracts\ShouldChunkRead;

class ProductImport implements ModelImport, ShouldChunkRead
{
    use Importable;

    protected int $chunkSize = 1000; // instead of config('import.chunk_size')

    // ...
}

Override chunkSize() instead when the size is computed rather than fixed.

Rows are written one upsert statement per row unless the importer implements ShouldBulkWrite, which groups a chunk's rows into statements of bulkSize() rows. The two contracts are independent. Importable supplies bulkSize() as well, reading a $bulkSize property and falling back to config('import.bulk_size'):

use Yossuf\Laravel\Import\Contracts\ShouldBulkWrite;

class ProductImport implements ModelImport, ShouldChunkRead, ShouldBulkWrite
{
    use Importable;

    protected int $chunkSize = 1000;

    protected int $bulkSize = 250; // instead of config('import.bulk_size')

    // ...
}

The model needs nothing from the package. Rows are written with Eloquent's upsert(), which bypasses model events, observers, attribute casts, set-mutators and $fillable/$guarded, but still fills in generated keys and timestamps:

Model's key	Who assigns it
Generated by the model (`HasUuids`, `HasUlids`, or any model overriding `newUniqueId()`)	Eloquent calls the model's `newUniqueId()` for each row whose `map()` leaves the key out
Auto-incrementing	The database. The engine never prefills it
Natural (e.g. a string code)	Your importer's `map()`, which returns the key as one of the attributes

map() receives only the columns rules() names, with the values as validated. A column the file has but the rules leave out never reaches map(), so a file cannot smuggle in an attribute the importer did not ask for, even when map() returns the row as is. Give every column you read a rule, if only ['present', 'nullable'].

map() has to return every uniqueBy() column. An importer that leaves one out fails the import with a LogicException naming the column.

Because casts and mutators do not run, map() has to return each value the way the column stores it. A hashed or encrypted cast, a json cast or a set-mutator on the model has no effect on an import, so a password mapped as is would be stored in plain text. Do in map() what the cast would have done:

use Illuminate\Support\Facades\Crypt;
use Illuminate\Support\Facades\Hash;

public function rules(): array
{
    return [
        'email' => ['required', 'email', 'max:255'],
        'password' => ['required', 'string', 'min:8'],
        'api_token' => ['present', 'nullable', 'string'],
        'roles' => ['present', 'nullable', 'string'],
    ];
}

public function map(array $row): array
{
    return [
        'email' => $row['email'],
        'password' => Hash::make($row['password']),
        'api_token' => $row['api_token'] === null ? null : Crypt::encryptString($row['api_token']),
        'roles' => $row['roles'] === null ? '[]' : json_encode(explode('|', $row['roles'])),
    ];
}

Reading the model back works as usual: the hashed, encrypted and array casts on User decode what map() stored, because the stored values are what User::create() would have written.

2. Run an import

use App\Imports\ProductImport;
use Yossuf\Laravel\Import\Facades\Import;

$record = Import::dispatch($file, new ProductImport());

Import::dispatch() is the package's only entry point. It hands the file to a queued job rather than reading it itself.

$file is trusted. The package hashes it, counts its lines, reads it and records the full path on the tracking record. Build the path yourself, for example from a Storage disk. Never build it from a user's original filename or a path a request supplies. The file is always parsed as CSV, whatever its extension.

3. How an import runs

Import::dispatch() returns straight away with a Waiting record. A queued read job (ReadImportFile) reads the file and import jobs (ImportModels) write the rows. The record settles when every import job is done:

$record = Import::dispatch($file, new ProductImport());

$record->state;         // ImportState::Waiting
$record->id;            // track it from here

The record also stores the file's SHA-256 hash, taken at dispatch. The read job recomputes it before reading and fails the import if it differs, so a file replaced between dispatch and processing is never imported under the old record. When the dispatching process cannot read the file, no hash is stored and the check is skipped.

Tracking Progress

Poll a fresh copy for progress:

$record->fresh()->progress();   // approximate while the file is read, exact once settled
$record->fresh()->state;        // Waiting → Processing → Success | Failed
$record->fresh()->rows_created;

Every row is counted as created, updated or skipped, so the three add up to rows_total once the import is done. Two rows sharing a uniqueBy() value both count, though the table ends up with one row.

The file is parsed once, streamed by one read job. Before streaming, the read job counts the file's lines in a single fast pass and stores that, minus the header, as a provisional rows_total, so progress() has a denominator from the first row. The estimate is an upper bound, since a quoted field spanning lines counts once per line. The exact count replaces it once the last row is read.

Chunks and Bulk Statements

The two optional contracts decide how rows get from the file to the database:

Importer implements	Reading	Writing
neither	row by row, each row imported inline as it is read	one statement per row
`ShouldChunkRead`	in chunks, each queued as an import job	one statement per row
`ShouldBulkWrite`	row by row, inline	one statement per row, since a job holds a single row
both	in chunks, each queued as an import job	statements of the bulk size

The chunk size has to be at least 1 and is resolved in this order:

the chunkSize argument to Import::dispatch()
the importer's chunkSize(), when it returns one. With Importable that is the importer's $chunkSize property
config('import.chunk_size'), 500 by default

The bulk size is resolved the same way, from the bulkSize argument, then bulkSize() or the $bulkSize property, then config('import.bulk_size'), and also has to be at least 1. Either argument is ignored for an importer without the matching contract.

Two things follow from the table:

ShouldBulkWrite on its own changes nothing. Every import job carries one row, so every statement writes one row.
A bulk size larger than the chunk size is capped by it. An import job never holds more than one chunk, so a chunk of 500 with a bulk size of 1000 is written in one statement of 500.

A row repeated in one file counts as created then updated when the two land in different statements, and as two created rows when one bulk statement holds both.

Import::dispatch($file, new ProductImport(), chunkSize: 1000, bulkSize: 250);

A chunked import's rows travel to the workers in the job payload, so a large chunk size with wide rows makes for large payloads. Lower the chunk size if your queue backend complains. The bulk size is the matching knob for the database, since a statement of many wide rows can exceed a driver's bound-parameter limit. An import without ShouldChunkRead never puts rows on the queue. It writes inside the read job, one row at a time.

Queue Requirements

The read job opens $file on the worker, not in the process that called Import::dispatch(). The path has to point at storage the worker can reach, such as a shared disk, and not at a request-local temp file. A path the worker cannot open settles the record Failed with the reader's "File does not exist" message.

Both jobs carry your ModelImport instance in their payload, so write it as a named class with no closures or open resources. An anonymous class, or one holding a file handle, fails at dispatch.

Import jobs of one import run one at a time, guarded by WithoutOverlapping, so the created/updated counts stay exact. Imports of different files still run in parallel.

Your app needs Laravel's job_batches table and a cache store that supports locks. Configure where the jobs go:

IMPORT_QUEUE_CONNECTION=redis
IMPORT_QUEUE=imports

Set IMPORT_QUEUE_CONNECTION=sync to run an import inline, as the package's own tests do. The returned record is then already settled, but a failing import rethrows after marking the record Failed, so a synchronous call needs a try/catch. On a real queue the same failure surfaces in the worker and the batch's catch() callback settles the record.

Cancelling an Import

The batch is named import:{id} after the record, so it can be found in job_batches and cancelled. A cancelled import stops reading, skips its remaining import jobs and settles the record Failed with "The import [id] was cancelled." Rows already written stay. The ImportFailed event then carries an ImportCancelledException, so a listener can tell a cancellation from other failures.

Failures

The record's error holds the exception message. A database failure is recorded by the driver's message alone. The SQL statement with its bound values and the connection details that Laravel appends are left out, so the tracking table never keeps a copy of the rows that failed. The driver's own message may still name the one value it rejected, as in a duplicate key error.

4. Listen for the outcome

The package fires an event when a record settles:

Event	Carries
`Yossuf\Laravel\Import\Events\ImportCompleted`	`$record`
`Yossuf\Laravel\Import\Events\ImportFailed`	`$record` and the `$exception` thrown

Both fire after the record's state and completed_at are saved, so a listener sees the final counts:

use Yossuf\Laravel\Import\Events\ImportCompleted;
use Yossuf\Laravel\Import\Events\ImportFailed;

Event::listen(ImportCompleted::class, function (ImportCompleted $event) {
    $event->record->rows_created;
});

Event::listen(ImportFailed::class, function (ImportFailed $event) {
    $event->record->error;          // the sanitized message
    $event->exception;              // the original exception
});

The events implement ShouldDispatchAfterCommit, so they wait for the transaction the settling code runs in. On the sync connection a failure fires ImportFailed once, from the rethrow, even though the batch's catch() callback runs first inside a transaction that is rolled back. A queued listener receives a fresh copy of the record. It should not rely on the exception serializing, since an exception holding closures in its trace does not.

License

Laravel Import is open-sourced software licensed under the Apache-2.0 license.

yossuf / laravel-import

Package info

Security

Statistics