rapide/laravel-queue-kafka

Kafka driver for Laravel Queue

Kafka Queue driver for Laravel

, _(*1)

Installation

1. Install librdkafka c library, _(*2)

   $ cd /tmp
   $ mkdir librdkafka
   $ cd librdkafka
   $ git clone https://github.com/edenhill/librdkafka.git .
   $ ./configure
   $ make
   $ make install
   

2. Install the php-rdkafka PECL extension, _(*3)

   $ pecl install rdkafka
   

3. a. Add the following to your php.ini file to enable the php-rdkafka extension extension=rdkafka.so, _(*4)

   b. Check if rdkafka is installed
   Note: If you want to run this on php-fpm restart your php-fpm first., _(*5)

   php -i | grep rdkafka, _(*6)

   Your output should look something like this, _(*7)

   rdkafka rdkafka support => enabled librdkafka version (runtime) => 1.0.0-RC2 librdkafka version (build) => 0.11.4.0, _(*8)

4. Install this package via composer using:, _(*9)

   composer require rapide/laravel-queue-kafka

5. Add LaravelQueueKafkaServiceProvider to providers array in config/app.php:, _(*10)

   Rapide\LaravelQueueKafka\LaravelQueueKafkaServiceProvider::class,

   If you are using Lumen, put this in bootstrap/app.php, _(*11)

   $app->register(Rapide\LaravelQueueKafka\LumenQueueKafkaServiceProvider::class);

6. Add these properties to .env with proper values:, _(*12)

   QUEUE_DRIVER=kafka

7. If you want to run a worker for a specific consumer group, _(*13)

   export KAFKA_CONSUMER_GROUP_ID="group2" && php artisan queue:work --sleep=3 --tries=3

   Explaination of consumergroups can be found in this article http://blog.cloudera.com/blog/2018/05/scalability-of-kafka-messaging-using-consumer-groups/, _(*14)

Usage

Once you completed the configuration you can use Laravel Queue API. If you used other queue drivers you do not need to change anything else. If you do not know how to use Queue API, please refer to the official Laravel documentation: http://laravel.com/docs/queues, _(*15)

Testing

Run the tests with:, _(*16)

bash vendor/bin/phpunit, _(*17)

Acknowledgement

This library is inspired by laravel-queue-rabbitmq by vyuldashev. And the Kafka implementations by Superbalist be sure to check those out., _(*18)

Contribution

You can contribute to this package by discovering bugs and opening issues. Please, add to which version of package you create pull request or issue., _(*19)

Supported versions of Laravel

Tested on: [5.4, 5.5, 5.6, 5.7], _(*20)

New version is underway

A version for Laravel 6,7,8 is underway. Also supporting the new RdKafka library, _(*21)