athlon1600 / serpscraper
PHP powered interface for querying the most popular Search Engines
Installs: 3 520
Dependents: 0
Suggesters: 0
Security: 0
Stars: 93
Watchers: 13
Forks: 38
Open Issues: 10
Requires
- ext-curl: *
- athlon1600/php-captcha-solver: ^2.0
- athlon1600/php-curl-client: ^1.1
Requires (Dev)
- phpstan/phpstan: ^1.12
- phpunit/phpunit: ^8.5.39 || ^9.5 || ^10.0
README
SerpScraper
The purpose of this library is to provide an easy, undetectable, and captcha resistant way to extract search results from popular search engines like Google and Bing.
Installation
The recommended way to install this is via Composer:
composer require athlon1600/serpscraper "^4.0"
Extracting Search Results From Google
<?php use SerpScraper\Engine\GoogleSearch; $page = 1; $google = new GoogleSearch(); // all available preferences for Google $google->setPreference('results_per_page', 100); //$google->setPreference('google_domain', 'google.lt'); //$google->setPreference('date_range', 'hour'); $results = array(); do { $response = $google->search("how to scrape google", $page); // error field must be empty otherwise query failed if(empty($response->error)){ $results = array_merge($results, $response->results); $page++; } else if($response->error == 'captcha'){ // read below break; } } while ($response->has_next_page);
Solve Google Search captchas automatically
For this to work, you will need to register for 2captcha.com services, and get an API key.
It is also highly recommended to use a proxy server.
Install a private proxy server on your own VPS here:
https://github.com/Athlon1600/useful#squid
<?php use SerpScraper\Engine\GoogleSearch; use SerpScraper\GoogleCaptchaSolver; $google = new GoogleSearch(); $browser = $google->getBrowser(); $browser->setProxy('PROXY:IP'); $solver = new GoogleCaptchaSolver($browser); while(true){ $response = $google->search('famous people born in ' . mt_rand(1500, 2020)); if ($response->error == 'captcha') { echo "Captcha detected!" . PHP_EOL; $temp = $solver->solveUsingTwoCaptcha($response, '2CAPTCHA_API_KEY', 90); if ($temp->status == 200) { echo "Captcha solved successfully!" . PHP_EOL; } else { echo 'Solving captcha has failed...' . PHP_EOL; } } else { echo "OK. "; } sleep(2); }
Extract Search Results from Bing
<?php use SerpScraper\Engine\BingSearch; $bing = new BingSearch(); $results = array(); for($page = 1; $page < 10; $page++){ $response = $bing->search("search bing using php", $page); if($response->error == false){ $results = array_merge($results, $response->results); } if($response->has_next_page == false){ break; } } var_dump($results);