sleimanx2/grawler

A guided html crawler with media meta extraction

Grawler

, _(*1)

Install

Via Composer, _(*2)

``` bash $ composer require sleimanx2/grawler, <sub>(*3)</sub>

## Basic Usage


##### getting the page dom

```php
require_once('vendor/autoload.php');

$client = new Bowtie\Grawler\Client();

$grawler = $client->download('http://example.com');

finding basic attributes

$grawler->title();

// provide a css path to find the attribute
$grawler->body($path = '.main-content');

// extracts meta keywords (array)
$grawler->keywords();

// extracts meta description 
$grawler->description();

finding media

$grawler->images('.content img');

$grawler->videos('iframe');

$grawler->audio('.audio iframe');

Resolving media attributes

In order resolve media attributes you need to load providers's configuration, <sub>(*4)</sub>

videos

Current video resolvers (youtube , vimeo), <sub>(*5)</sub>

// resolve all videos at once 
$videos = $grawler->videos('iframe')->resolve();

then you can access videos attributes as follow, <sub>(*6)</sub>

foreach($videos as $video)
{
  $video->id; // the video provider id
  $video->title;
  $video->description;
  $video->url;
  $video->embedUrl;
  $video->images; // Collection of Image instances
  $video->author;
  $video->authorId;
  $video->duration;
  $video->provider; //video source
}

you can also resolve videos individually as follow, <sub>(*7)</sub>

$videos = $grawler->videos('iframe')

foreach($videos as $video)
{
  $video->resolve();
  $video->title;
  //...
}

audio

Current video resolvers (soundcloud), <sub>(*8)</sub>

// resolve all audio at once 
$audio = $grawler->audio('.audio iframe')->resolve();

then you can access videos attributes as follow, <sub>(*9)</sub>

foreach($audio as $track)
{
  $track->id; // the video provider id
  $track->title;
  $track->description;
  $track->url;
  $track->embedUrl;
  $track->images; // Collection of cover photo instances
  $track->author;
  $track->authorId;
  $track->duration;
  $track->provider; //video source
}

you can also resolve audio individually as follow, <sub>(*10)</sub>

$track = $grawler->track('.audio iframe')

foreach($audio as $track)
{
  $track->resolve();
  $track->title;
  //...
}

Resolving page urls

$links = $grawler->links('.main thumb a')

foreach($links as $link)
{
  print $link
  //or
  print $link->uri
  //or
  print $link->getUri()
}

Configuration

Client Config

Set user agent

$client->agent('Googlebot/2.1')->download('http://example.com');

Recomended : http://webmasters.stackexchange.com/questions/6205/what-user-agent-should-i-set, <sub>(*11)</sub>

Set request auth

$client->auth('me', '**')

you can change the auth type as follow, <sub>(*12)</sub>

$client->auth('me', '**', $type = 'basic');

Set request method

$client->method('post');

Grawler config

By default the grawler tries to access those environment variables, <sub>(*13)</sub>

GRAWLER_YOUTUBE_KEY

GRAWLER_VIMEO_KEY
GRAWLER_VIMEO_SECRET

GRAWLER_SOUNDCLOUD_KEY
GRAWLER_SOUNDCLOUD_SECRET

if you don't use env vars you can load configuration as follow., <sub>(*14)</sub>

$config = [
  'youtubeKey'   =>'',
  'soundcloudKey'=>''

  'vimeoKey'    => '',
  'vimeoSecret' => '',

  'soundcloudKey'    => '',
  'soundcloudSecret' => '',
];

$grawler->loadConfig($config);

Testing

``` bash $ phpunit --testsuite unit, <sub>(*15)</sub>

``` bash
$ phpunit --testsuite integration

NB: you should set your ptoviders key (youtube,vimeo,soundcloud...) to run integration tests, <sub>(*16)</sub>

Contributing

Please see CONTRIBUTING, <sub>(*17)</sub>

Security

If you discover any security related issues, please email sleiman@bowtie.land instead of using the issue tracker., <sub>(*18)</sub>

License

The MIT License (MIT). Please see License File for more information., <sub>(*19)</sub>