2017 © Pedro Peláez
 

library php-mapreduce

A local implementation of the map/reduce strategy in PHP

image

jotaelesalinas/php-mapreduce

A local implementation of the map/reduce strategy in PHP

  • Sunday, October 29, 2017
  • by jotaelesalinas
  • Repository
  • 2 Watchers
  • 4 Stars
  • 23 Installations
  • PHP
  • 0 Dependents
  • 0 Suggesters
  • 4 Forks
  • 0 Open issues
  • 9 Versions
  • 10 % Grown

The README.md

php-mapreduce

Latest Version on Packagist ![Software License][ico-license] Build Status ![Total Downloads][ico-downloads], (*1)

PHP PSR-4 compliant library to easily do non-distributed locla map-reduce., (*2)

Install

Via Composer, (*3)

``` bash $ composer require jotaelesalinas/php-mapreduce, (*4)


## Usage A very simple example were we have a CSV with the full order history of an hypothetical online shop and we want to know the average order value. ``` php use JLSalinas\MapReduce\MapReduce; use JLSalinas\RWGen\Readers\Csv; $mapper = function($order) { return [ 'orders' => 1, 'revenue' => $order['total_amount'] ]; }; $reducer = function ($carry, $item) { if ( is_null($carry) ) { $item['avg_order_value'] = $item['revenue'] / $item['orders']; return $item; } $orders = $carry['orders'] + $item['orders']; $revenue = $carry['revenue'] + $item['revenue']; $avg_order_value = $revenue / $orders; return compact('orders', 'revenue', 'avg_order_value'); }; $mapreducer = (new MapReduce(new Csv('/path/to/file.csv'))) ->map($mapper) ->reduce($reducer) ->run();

Now an example where we also read from a CSV with the order history of an online shop, writing the output to another CSV, and we want to know for each customer: - date of the last order - number of orders since the beginning - amount spent since the beginning - average order value since the beginning - number of orders in the last 12 months - amount spent in the last 12 months - average order value in the last 12 months, (*5)

``` php use JLSalinas\MapReduce\MapReduce; use JLSalinas\RWGen\Readers\Csv; use JLSalinas\RWGen\Writers\Csv;, (*6)

$mapper = function($order) { return [ 'customer_id' => $order['customer_id'], 'date_last_order' => $order['date'], 'orders' => 1, 'orders_last_12m' => strtotime($order['date']) > strtotime('-12 months') ? 1 : 0, 'revenue' => $order['total_amount'], 'revenue_last_12m' => strtotime($order['date']) > strtotime('-12 months') ? $order['total_amount'] : 0 ]; };, (*7)

$reducer = function ($carry, $item) { if ( is_null($carry) ) { $item['avg_revenue'] = $item['revenue'] / $item['orders']; $item['avg_revenue_last_12m'] = $item['orders_last_12m'] ? $item['revenue_last_12m'] / $item['orders_last_12m'] : 0; return $item; }, (*8)

$date_last_order      = max($carry['date_last_order'], $item['date_last_order']);
$orders               = $carry['orders'] + $item['orders'];
$orders_last_12m      = $carry['orders_last_12m'] + $item['orders_last_12m'];
$revenue              = $carry['revenue'] + $item['revenue'];
$revenue_last_12m     = $carry['revenue_last_12m'] + $item['revenue_last_12m'];
$avg_revenue          = $revenue / $orders;
$avg_revenue_last_12m = $orders_last_12m > 0 ? $revenue_last_12m / $orders_last_12m : 0;

return compact('date_last_order', 'orders', 'orders_last_12m', 'revenue', 'revenue_last_12m', 'avg_revenue', 'avg_revenue_last_12m');

};, (*9)

$mapreducer = (new MapReduce(new Csv('/path/to/input_file.csv'))) ->map($mapper) ->reduce($reducer, true) ->writeTo(new Csv('/path/to/output_file.csv')) ->run();, (*10)


You can see more elaborated examples under the folder [docs](docs). ## Change log Please see [CHANGELOG](CHANGELOG.md) for more information what has changed recently. ## Testing ``` bash $ composer test

Contributing

Please see CONTRIBUTING and CONDUCT for details., (*11)

Security

If you discover any security related issues, please DM me to @jotaelesalinas instead of using the issue tracker., (*12)

To do

  • [ ] Tests events in MapReduce
  • [ ] Add docs
  • [ ] Insurance example
    • [ ] adapt to new library
    • [ ] add insured values
    • [ ] improve kml output (info, markers)
  • [ ] (Enhancement) withBuffer(int $max_size) to allow mapping and reducing in batches
    • [ ] (Enhancement) Multithread (requires pthreads)
      • [ ] (Enhancement) Pipelining: map while reading, reduce while mapping
  • [ ] Mention that it is possible to work both with local and cloud data by implementing the right Reader/Writer, possibly using Flysystem by Frank de Jonge.
  • [ ] Move this to-do list to Issues
  • [ ] Create milestones in GitHub for: sequential (v1.0), buffered (v1.1), multithreaded (v1.2), pipelined (v1.3).

Credits

License

The MIT License (MIT). Please see License File for more information., (*13)

The Versions

29/10 2017

dev-master

9999999-dev https://github.com/jotaelesalinas/php-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-mapreduce

29/10 2017

v1.0.5

1.0.5.0 https://github.com/jotaelesalinas/php-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-mapreduce

30/09 2017

v1.0.4

1.0.4.0 https://github.com/jotaelesalinas/php-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-mapreduce

31/08 2016

v1.0.2

1.0.2.0 https://github.com/jotaelesalinas/php-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-mapreduce

30/08 2016

v1.0.1

1.0.1.0 https://github.com/jotaelesalinas/php-local-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map local mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-local-mapreduce

29/08 2016

v1.0.0

1.0.0.0 https://github.com/jotaelesalinas/php-local-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map local mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-local-mapreduce

28/08 2016

v0.2.1

0.2.1.0 https://github.com/jotaelesalinas/php-local-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map local mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-local-mapreduce

26/08 2016

v0.2.0

0.2.0.0 https://github.com/jotaelesalinas/php-local-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map local mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-local-mapreduce

21/08 2016

v0.1.0

0.1.0.0 https://github.com/jotaelesalinas/php-local-mapreduce

A local implementation of the map/reduce strategy in PHP

  Sources   Download

MIT

The Requires

 

The Development Requires

php map local mapreduce reduce jotaelesalinas jlsalinas map-reduce non-distribted php-local-mapreduce