innmind / robots-txt
Robots.txt parser
6.2.0
2023-11-01 13:50 UTC
Requires
- php: ~8.2
- innmind/http-transport: ~7.0
- innmind/immutable: ~4.13|~5.0
- innmind/url: ~4.1
Requires (Dev)
- innmind/coding-standard: ~2.0
- phpunit/phpunit: ~9.0
- vimeo/psalm: ~5.15
README
Robots.txt parser
Installation
composer require innmind/robots-txt
Usage
use Innmind\RobotsTxt\Parser; use Innmind\OperatingSystem\Factory; use Innmind\Url\Url; $os = Factory::build(); $parse = Parser::of( $os->remote()->http(), 'My user agent', ); $robots = $parse(Url::of('https://github.com/robots.txt'))->match( static fn($robots) => $robots, static fn() => throw new \RuntimeException('robots.txt not found'), ); $robots->disallows('My user agent', Url::of('/humans.txt')); //false $robots->disallows('My user agent', Url::of('/any/other/url')); //true
Note: Here only the path /humans.txt
is allowed because by default github disallows any user agent to crawl there website except for this file.