2017 © Pedro Peláez
 

library jmem

Break up large json files into reasonable sizes.

image

mschuett/jmem

Break up large json files into reasonable sizes.

  • Monday, February 22, 2016
  • by michaeljs1990
  • Repository
  • 3 Watchers
  • 10 Stars
  • 144 Installations
  • PHP
  • 1 Dependents
  • 0 Suggesters
  • 1 Forks
  • 0 Open issues
  • 1 Versions
  • 0 % Grown

The README.md

Jmem

Iterate through large JSON arrays without eating up all your memory., (*1)

Example Use

To start using jmem add the following line to your composer.json file., (*2)

{
    "require": {
        "mschuett/jmem": "dev-master"
    }
}

Next setup your autloaded and away you go., (*3)

<?php require 'vendor/autoload.php';

$gen = new Jmem\JsonLoader("Hangouts.json", "conversation_state");

foreach($gen->parse()->start() as $obj) {

    $obj->stream;

}

About

Jmem was written to parse huge JSON files when trying to put google hangouts json data a 200MB+ file into a database. PHP is likely not the best tool for this job however when it's the only thing available it does just fine. It currently takes about 1.5 minutes to break up a 200MB file so running this in the background after an upload is your best bet. A generator is used as to not take up large chunks of memory at any time., (*4)

Documentation

I have comented the code extreamly well so it is very easy to look through the source code and make changes. With that said here is the main JsonLoader Class., (*5)

/**
 * $file is the path to the file that you would like to parse though.
 * If the file does not exist an exception will be thrown. Element is
 * the array of objects you would like to have broken up and returned
 * to you.
 * 
 * @param String $file
 * @param String $element
 * @param int $bytes (1024 default)
 */

The generator returns a JSON object that gives you access to the following. Stream contains the full json object and number is the place of of the object in the array starting at 1., (*6)

class JsonObject {

    public $number = 0;

    public $stream = "";

    public function __construct($stream, $num) {
        $this->stream = $stream;
        $this->number = $num;
    }

}

Contribute

Please feel free to post bugs you may find or submit a pull request. I love feedback good and bad!, (*7)

The Versions

22/02 2016

dev-master

9999999-dev

Break up large json files into reasonable sizes.

  Sources   Download

MIT

The Requires

  • php >=5.5.0

 

by Michael Schuett