disqontrol/disqontrol

A user-friendly framework for running background jobs in any language.

Disqontrol

![Software License][ico-license] ![Total Downloads][ico-downloads], _(*1)

Introduction

Disqontrol is a user-friendly framework for running background jobs in any language., _(*2)

Imagine your user has just uploaded a large picture and you want to create four other pictures from it. But the pictures are not needed immediately and you don't want to keep the user waiting for the server response., _(*3)

Instead of processing the picture right now and delaying the response to the user, you create a background job and respond to the user without waiting for the result. The background job will be picked up by another program that will process the picture without delaying your main application., _(*4)

There are five steps in this process:, _(*5)

1. Creating a background job
2. Storing it
3. Picking it up from the storage
4. Processing it
5. Handling failed jobs

Where does Disqontrol come in?, _(*6)

Ad 1. Disqontrol helps you create a new background job, either from PHP by using the Disqontrol producer, or from a command line. You can also add the job with your own preferred library., _(*7)

Adding a job in PHP: ``` php $producer->add(new Job('example-pic.png', 'resize-pic'));, _(*8)

Adding a job from the command line:
``` bash
disqontrol addjob resize-pic '"example-pic.png"'

Ad 2. The job is stored in Disque, a specialized job queue created by the author of Redis., _(*9)

Ad 3. The job is picked up by a Disqontrol consumer, a long-running process that listens to Disque, picks up jobs coming in and decides who should process them., _(*10)

Configuration: ``` yaml queues: 'resize-pic': worker: type: isolated_php_worker name: 'resize_pic_worker', _(*11)

Ad 4. The code for processing a job - the worker - is ultimately your
responsibility, but Disqontrol provides a helpful scaffolding to write your
own workers, so you can just concentrate on the processing itself. Just follow
the interface.

The simplest PHP worker for resizing pictures:
``` php
class ResizePicWorker implements WorkerInterface
{
    public function process(JobInterface $job)
    {
        $picFileName = $job->getBody();
        // Concrete implementation left out
        return $this->resize($picFileName);
    }
}

Ad 5. It's important to decide what to do with failed jobs. Even though the jobs are running in the background, some of them must be processed, some make only sense for a certain amount of time and some of them are optional. Disqontrol helps you configure failure handling for each job type individually, as well as set a reasonable default behavior., _(*12)

Features

Disqontrol is written in PHP and uses Disque as its underlying job queue., _(*13)

With Disqontrol you get the following features:, _(*14)

• Process jobs in any programming language
• Schedule jobs and run them regularly
• Write PHP workers with all the scaffolding already in place
• Configure the lifecycle of a job in detail
• Handle failures robustly
• Process new jobs immediately for development and debugging
• Run multiple jobs from one queue in parallel
• Switch automatically to the best Disque node

Workers can be called via a command line and can therefore be written in any language. The library also provides convenient scaffolding for workers written in PHP., _(*15)

The goal of Disqontrol is to be a user-friendly, clean and robust tool., _(*16)

Disqontrol follows semantic versioning., _(*17)

Documentation

• Introduction
• Features
• Terminology
• Requirements
• Installation
• Getting started
• Configuration
• Adding new jobs
• Processing jobs
• Scheduled jobs
• What happens with failed jobs?
• Debugging jobs in a synchronous mode

Why Disque?

There are many general message queues - RabbitMQ, ZeroMQ, IronMQ, Amazon SQS. Redis also has a queue functionality. Some libraries even offer databases as queue backends., _(*18)

General message queues treat a job just like a message and the job workflow must be simulated in libraries., _(*19)

Each queue implementation has different features and weaknesses. Queue libraries usually allow you to choose from different backends and try to make up for missing features in the code. This leads to the problem of a common denominator, unnecessary complexity and doesn't allow one to use the features of one queue fully., _(*20)

That's why we have decided to use just one queue - Disque - written by the developer of Redis., _(*21)

For the above mentioned reasons, we have no plans to support other queues. Instead we want to fully use all features of Disque., _(*22)

Roadmap

The following features are planned for the 1.0 release:, _(*23)

• [x] Consumers pick up jobs and call workers
• [x] Supervisor keeps Consumers alive
• [x] Consumers quit gracefully
• [x] Isolated and inline PHP workers
• [x] Command-line workers
• [x] Failing jobs re-enqueued with exponential backoff
• [x] Disqontrol has access to a PHP app environment
• [x] Scheduled repeated jobs
• [x] Synchronous mode
• [ ] HTTP workers (a job can be processed by an HTTP call)
• [ ] Autoscaling of consumers ("just" the algorithm is missing, see Disqontrol\Consumer\Autoscale\PredictiveAutoscaling)
• [ ] Using custom failure strategies (see Disqontrol\Dispatcher\JobDispatcher and Disqontrol\Dispatcher\Failure\FailureStrategyCollection)
• [ ] The --config parameter (for installations not using the bootstrap file, eg. teams not writing PHP code)

Change log

Please see the CHANGELOG for more information what has changed recently., _(*24)

Contributing

Please see CONTRIBUTING for details., _(*25)

Credits

• All Contributors
• Big thanks to Antirez for Disque
• Big thanks to Mariano for Disque-php
• Thanks to Martin Patera for kicking off the work

Alternatives

For similar libraries in PHP, have a look at, _(*26)

• Bernard
• PHP-Queue
• php-resque
• php-enqueue

For libraries in other languages, have a look at, _(*27)

• Resque in Ruby
• Celery in Python

License

The MIT License (MIT). Please see the License for more information., _(*28)