linkorb/pipeline

Pipeline processing library

Pipeline

Pipeline is a library + CLI tool to load programmable pipelines, execute jobs and get detailed results per stage for reporting/debugging., _(*1)

Concepts

• Pipeline: A sequential list of stages
• Stage: An command template with a unique name within the pipeline.
• Job: A job that will be executed on a pipeline with an optional set of input variables
• JobResult: The output of a Job that has been executed on a pipeline, contains status and a StageResult for each stage in the pipeline
• StageResult: The exact command that was executed, it's exitcode, stdout and stderr.

CLI examples

Load a pipeline YML file and execute it with input variable topic=technology, _(*2)

bin/pipeline run examples/bbc-news-demo.pipeline.yml -d topic=technology

Load a pipeline YML file and execute it with input from STDIN, _(*3)

bin/pipeline run examples/bbc-news-stdin-demo.pipeline.yml < examples/bbc-news-demo.rss.xml

Output a JSON result of the JobResult + StageResults to result.json, _(*4)

bin/pipeline run -o result.json examples/bbc-news-demo.pipeline.yml -d topic=technology

Load a pipeline YML file and intentially cause an error by requesting a non-existant topic. This will output the errors., _(*5)

bin/pipeline run examples/bbc-news-demo.pipeline.yml -d topic=this-topic-does-not-exist

You can suppress output and debugging output by using --quiet. You can use this in combination with -o to read the details from a .json file., _(*6)

In all cases the exit code of the pipeline command will match the exit code of the last stage result: 0 on success, or other on failure., _(*7)

Library example

Please refer to examples/example-code.php on how to use the pipeline package as a library., _(*8)

Configuration through environment variables

pipeline will use environment variables in the inputs of pipelines. This way you don't need to define all required variables using -d., _(*9)

When pipeline runs, it first checks if a .env file exists in the current directory and update the environment variabels based on it's contents., _(*10)

License

MIT. Please refer to the license file for details., _(*11)

Brought to you by the LinkORB Engineering team

Check out our other projects at linkorb.com/engineering., _(*12)

Btw, we're hiring!, _(*13)