katsana/dusk-crawler

Web Crawler using Laravel Dusk

v0.1.6 2020-10-21 13:19 UTC

Requires

• php: >=7.2
• laravel/dusk: ^5.11 || ^6.0
• orchestra/canvas-core: ^4.7 || ^5.0 || ^6.0
• orchestra/dusk-updater: ^1.2
• react/promise: ^2.7
• symfony/dom-crawler: ^4.3 || ^5.0

Requires (Dev)

• orchestra/canvas: ^4.5 || ^5.0 || ^6.0
• orchestra/testbench: ^4.5 || ^5.0 || ^6.0

MIT

• KATSANA Developer
• Mior Muhammad Zaki

README

Laravel Dusk enables developer to run browser automation but it does lack the ability to navigate actions based on response received on the browser. If you need to handle failure you have to wait until the timeout expired and handle the generic exception.

Dusk Crawler solve this by adding inspectUsing() method to allow developer to inspect for success or fail status using Promise by ReactPHP.

Installation

Dusk Crawler can be installed via composer:

composer require "katsana/dusk-crawler"

Usages

Dusk Crawler just add two new macros to Laravel\Dusk\Browser:

Example

Let say you want to crawl Packagist to search for some package, and the input is dynamic.

use DuskCrawler\Dusk;
use DuskCrawler\Inspector;
use DuskCrawler\Exceptions\InspectionFailed;
use Laravel\Dusk\Browser;

function searchPackagist(string $packagist) {
    $dusk = new Dusk('search-packagist');

    $dusk->headless()->disableGpu()->noSandbox();
    $dusk->userAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36');

    $dusk->start();

    $dusk->browse(function ($browser) use ($packagist) {
        $browser->visit('https://packagist.org/');

        $promise = $browser->type('search_query[query]', $packagist, '{enter}')
            ->inspectUsing(15, function (Browser $browser, Inspector $inspector) {
                $searchList = $browser->resolver->findOrFail('.search-list');

                if (! $searchList->isDisplayed() || $searchList->getText() == '') {
                    // result not ready, just return false.
                    return false;
                }

                if ($searchList->getText() == 'No packages found.') {
                    return $inspector->abort('No packages found!');
                }

                return $inspector->resolve();
            });

        $promise->then(function ($browser) {
            // Crawl the page on success.
            $packages = $browser->crawler()
              ->filter('div.package-item')->each(function ($div) {
                return $div->text();
            });
      
            dump($packages);
        })->otherwise(function (InspectionFailed $exception) {
            // Handle abort state.
            dump("No result");
        })->done();
    });

    $dusk->stop();
}

searchPackagist('dusk-crawler');

Dusk::closeAll();