crwlr/robots-txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

v1.1.2 2025-01-27 17:32 UTC

Requires

• php: ^8.0
• crwlr/url: ^1.0|^2.0

Requires (Dev)

• friendsofphp/php-cs-fixer: ^3.57
• mockery/mockery: ^1.4
• phpstan/phpstan: ^1.1
• phpunit/phpunit: ^9.0
• sempro/phpunit-pretty-print: ^1.4

MIT

• Christian Olear

• parser
• scraper
• spider
• crawler
• bot
• robots.txt
• robots
• protocol
• standard
• scraping
• bots
• exclusion
• crwlr

README

Robots Exclusion Standard/Protocol Parser

for Web Crawling/Scraping

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Documentation

You can find the documentation at crwlr.software.

Contributing

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).