ddliu/spider

Light weight spider for the web.

Spider

A flexible spider in PHP., _(*1)

Concepts

A spider contains many processors called pipes, you can pass as many tasks as you like to the spider, each task go through these pipes and get processed., _(*2)

Installation

composer require ddliu/spider

Requirements

• PHP5.3+
• curl(RequestPipe)

Dependencies

See composer.json., _(*3)

Usage

use ddliu\spider\Spider;
use ddliu\spider\Pipe\NormalizeUrlPipe;
use ddliu\spider\Pipe\RequestPipe;
use ddliu\spider\Pipe\DomCrawlerPipe;

(new Spider())
    ->pipe(new NormalizeUrlPipe())
    ->pipe(new RequestPipe())
    ->pipe(new DomCrawlerPipe())
    ->pipe(function($spider, $task) {
        $task['$dom']->filter('a')->each(function($a) use ($task) {
            $href = $a->attr('href');
            $task->fork($href);
        })
    })
    // the entry task
    ->addTask('http://example.com')
    ->run()
    ->report();

Find more examples in examples folder., _(*4)

Spider

The Spider class., _(*5)

Options

• limit: maxmum tasks to run

Methods

• pipe($pipe): add a pipe
• addTask($task): add a task
• run(): run the spider
• report(): write report to log

Task

A task contains the data array and some helper functions., _(*6)

The Task class implements ArrayAccess interface, so you can access data like array., _(*7)

Methods

• fork($task): add a sub task to the spider
• ignore(): ignore the task

Pipes

Pipes define how each task being processed., _(*8)

A pipe can be a function:, _(*9)

function($spider, $task) {}

Or extends the BasePipe:, _(*10)

use ddliu\spider\Pipe\BasePipe;

class MyPipe extends BasePipe {
    public function run($spider, $task) {
        // process the task...
    }
}

Useful Pipes

NormalizeUrlPipe

Normalize $task['url']., _(*11)

new NormalizeUrlPipe()

RequestPipe

Start an HTTP request with $task['url'] and save the result in $task['content']., _(*12)

new RequestPipe(array(
    'useragent' => 'myspider',
    'timeout' => 10
));

FileCachePipe

Cache a pipe (e.g. RequestPipe)., _(*13)

$requestPipe = new RequestPipe();
$cacheForReqPipe = new FileCachePipe($requestPipe, [
    'input' => 'url',
    'output' => 'content',
    'root' => '/path/to/cache/root',
]);

RetryPipe

Retry on failure., _(*14)

$requestPipe = new RequestPipe();
$retryForReqPipe = new RetryPipe($requestPipe, [
    'count' => 10,
]);

DomCrawlerPipe

Create a DomCrawler from $task['content']. Access it with $task['$dom'] in following pipes., _(*15)

ReportPipe

Report every 10 minutes., _(*16)

new ReportPipe(array(
    'seconds' => 600
))

Logging

$spider->logger is an instance of Monolog\Logger. You can add logging handlers to it before start:, _(*17)

use Monolog\Handler\StreamHandler;

$spider->logger->pushHandler(new StreamHandler('path/to/your.log', Logger::WARNING));

TODO/Ideas

• Real world examples.
• Running tasks concurrently.(With pthread?)

Alternate

Use golang version for better performance!, _(*18)