Wallogit.com
2017 © Pedro Peláez
Symfony Bundle - Use file watchers with MariaDb Columnstore CpImport utility
MariaDb ColumnStore is a very powerful, column-oriented MySQL storage, powerful for analytics and Big Data., (*1)
It has the drawback of extremely slow write operations, but comes with cpimport, a binary that can load CSV files with millions of rows into the table of your choice, in a few seconds., (*2)
This Symfony bundle enables a file watcher that will look into a directory, pick up new CSV files when they come, run cpimport and delete them., (*3)
[!IMPORTANT]
This repository is no longer maintained and may be removed in a near future. You may consider creating a fork if you still require it., (*4)
composer require bentools/cpimport-bundle:1.0.x-dev, (*5)
Add the bundle to your kernel. Then, map your directories to database/tables like this:, (*6)
# config/packages/cpimport.yaml (or app/config.yml in Symfony 3)
cpimport:
cpimport_bin: /usr/local/mariadb/columnstore/bin/cpimport
watch:
/path/to/csv_sales:
database: my_shop
table: sales_day
options:
delimiter: ','
enclosure: '"'
timeout: 60
To automatically process new files in configured directories:, (*7)
php bin/console cpimport:watch
To manually process a CSV file through cpimport:, (*8)
php bin/console cpimport:run my_shop sales_day path/to/file.csv --delimiter=','
Symfony Bundle - Use file watchers with MariaDb Columnstore CpImport utility
MIT
php symfony file watcher mariadb columnstore cpimport