bentools / cpimport-bundle
Symfony Bundle - Use file watchers with MariaDb Columnstore CpImport utility
Requires
- php: ^7.1.3
- symfony/config: ~3.0||~4.0
- symfony/console: ~3.0||~4.0
- symfony/dependency-injection: ~3.0||~4.0
- symfony/http-kernel: ~3.0||~4.0
- symfony/lock: ~3.0||~4.0
- symfony/process: ~3.0||~4.0
- symfony/yaml: ~3.0||~4.0
Requires (Dev)
- symfony/dotenv: ~4.0
- symfony/framework-bundle: ~3.0||~4.0
- symfony/var-dumper: ~4.0
This package is auto-updated.
Last update: 2024-12-07 08:13:52 UTC
README
MariaDb ColumnStore is a very powerful, column-oriented MySQL storage, powerful for analytics and Big Data.
It has the drawback of extremely slow write operations, but comes with cpimport
, a binary that can load CSV files with millions of rows into the table of your choice, in a few seconds.
This Symfony bundle enables a file watcher that will look into a directory, pick up new CSV files when they come, run cpimport
and delete them.
Important
This repository is no longer maintained and may be removed in a near future. You may consider creating a fork if you still require it.
Installation
composer require bentools/cpimport-bundle:1.0.x-dev
Configuration
Add the bundle to your kernel. Then, map your directories to database/tables like this:
# config/packages/cpimport.yaml (or app/config.yml in Symfony 3) cpimport: cpimport_bin: /usr/local/mariadb/columnstore/bin/cpimport watch: /path/to/csv_sales: database: my_shop table: sales_day options: delimiter: ',' enclosure: '"' timeout: 60
Usage
To automatically process new files in configured directories:
php bin/console cpimport:watch
To manually process a CSV file through cpimport
:
php bin/console cpimport:run my_shop sales_day path/to/file.csv --delimiter=','