athlon1600/serpscraper

PHP powered interface for querying the most popular Search Engines

, _(*1)

SerpScraper

The purpose of this library is to provide an easy, undetectable, and captcha resistant way to extract search results from popular search engines like Google and Bing., _(*2)

Installation

The recommended way to install this is via Composer:, _(*3)

composer require athlon1600/serpscraper "^4.0"

Extracting Search Results From Google

<?php

use SerpScraper\Engine\GoogleSearch;

$page = 1;

$google = new GoogleSearch();

// all available preferences for Google
$google->setPreference('results_per_page', 100);
//$google->setPreference('google_domain', 'google.lt');
//$google->setPreference('date_range', 'hour');

$results = array();

do {

    $response = $google->search("how to scrape google", $page);

    // error field must be empty otherwise query failed
    if(empty($response->error)){

        $results = array_merge($results, $response->results);
        $page++;

    } else if($response->error == 'captcha'){

        // read below
        break;
    }

} while ($response->has_next_page);

Solve Google Search captchas automatically

For this to work, you will need to register for 2captcha.com services, and get an API key. It is also highly recommended to use a proxy server.
Install a private proxy server on your own VPS here:
https://github.com/Athlon1600/useful#squid, _(*4)

<?php

use SerpScraper\Engine\GoogleSearch;
use SerpScraper\GoogleCaptchaSolver;

$google = new GoogleSearch();

$browser = $google->getBrowser();
$browser->setProxy('PROXY:IP');

$solver = new GoogleCaptchaSolver($browser);

while(true){
    $response = $google->search('famous people born in ' . mt_rand(1500, 2020));

    if ($response->error == 'captcha') {

        echo "Captcha detected!" . PHP_EOL;

        $temp = $solver->solveUsingTwoCaptcha($response, '2CAPTCHA_API_KEY', 90);

        if ($temp->status == 200) {
            echo "Captcha solved successfully!" . PHP_EOL;
        } else {
            echo 'Solving captcha has failed...' . PHP_EOL;
        }

    } else {
        echo "OK. ";
    }

    sleep(2);
}

Extract Search Results from Bing

<?php

use SerpScraper\Engine\BingSearch;

$bing = new BingSearch();
$results = array();

for($page = 1; $page < 10; $page++){

    $response = $bing->search("search bing using php", $page);
    if($response->error == false){
        $results = array_merge($results, $response->results);
    }

    if($response->has_next_page == false){
        break;
    }
}

var_dump($results);