webhubworks/site-crawler

A straightforward site crawler

1.0.4 2024-11-21 06:14 UTC

Requires

• php: ^8.2.0
• ext-dom: *
• illuminate/http: ^11.29
• illuminate/validation: ^11.29
• spatie/url: ^2.4

Requires (Dev)

• laravel-zero/framework: ^11.0.0
• laravel/pint: ^1.15.2
• mockery/mockery: ^1.6.11
• pestphp/pest: ^2.34.7

MIT

• Johannes Ahrndt

• framework
• cli
• console
• laravel
• Laravel Zero

README

Use this site crawler as a quick way to crawl any website. This is useful to detect any slow pages or pages with HTTP errors.

Please use this crawler responsibly. Do not use it to crawl websites that you do not own or have permission to crawl.

Installation

composer global require webhubworks/site-crawler

Usage

Use the help: site-crawler --help

Example: site-crawler https://example.com --limit=50 --basic-auth=user:pass --exclude=action,imprint

Roadmap

• Add support for websites containing links in JS generated markup
• Run requests in parallel