Arachnid Web Crawler

This library will crawl all unique internal links found on a given website up to a specified maximum page depth., _(*1)

This library is based on the original blog post by Zeid Rashwani here:, _(*2)

http://zrashwani.com/simple-web-spider-php-goutte, _(*3)

Josh Lockhart adapted the original blog post's code (with permission) for Composer and Packagist and updated the syntax to conform with the PSR-2 coding standard., _(*4)

, _(*5)

How to Install

You can install this library with Composer. Drop this into your composer.json manifest file:, _(*6)

{
    "require": {
        "zrashwani/arachnid": "dev-master"
    }
}

Then run composer install., _(*7)

Getting Started

Here's a quick demo to crawl a website:, _(*8)

<?php
require 'vendor/autoload.php';

$url = 'http://www.example.com';
$linkDepth = 3;
// Initiate crawl    
$crawler = new \Arachnid\Crawler($url, $linkDepth);
$crawler->traverse();

// Get link data
$links = $crawler->getLinks();
print_r($links);

Advanced Usage:

There are other options you can set to the crawler:, _(*9)

Set additional options to underlying guzzle client, by specifying array of options in constructor or passing it to setCrawlerOptions:, _(*10)

<?php
    //third parameter is the options used to configure guzzle client
    $crawler = new \Arachnid\Crawler('http://github.com',2, 
                             ['auth'=>array('username', 'password')]);

    //or using separate method `setCrawlerOptions`
    $options = array(
        'curl' => array(
            CURLOPT_SSL_VERIFYHOST => false,
            CURLOPT_SSL_VERIFYPEER => false,
        ),
        'timeout' => 30,
        'connect_timeout' => 30,
    );

    $crawler->setCrawlerOptions($options);

You can inject a PSR-3 compliant logger object to monitor crawler activity (like Monolog):, _(*11)

<?php    
$crawler = new \Arachnid\Crawler($url, $linkDepth); // ... initialize crawler   

//set logger for crawler activity (compatible with PSR-3)
$logger = new \Monolog\Logger('crawler logger');
$logger->pushHandler(new \Monolog\Handler\StreamHandler(sys_get_temp_dir().'/crawler.log'));
$crawler->setLogger($logger);
?>

You can set crawler to visit only pages with specific criteria by specifying callback closure using filterLinks method:, _(*12)

<?php
//filter links according to specific callback as closure
$links = $crawler->filterLinks(function($link){
                    //crawling only blog links
                    return (bool)preg_match('/.*\/blog.*$/u',$link); 
                })
                ->traverse()
                ->getLinks();

How to Contribute

Fork this repository
Create a new branch for each feature or improvement
Apply your code changes along with corresponding unit test
Send a pull request from each feature branch

It is very important to separate new features or improvements into separate feature branches, and to send a pull request for each branch. This allows me to review and pull in new features or improvements individually., _(*13)

All pull requests must adhere to the PSR-2 standard., _(*14)