fkrzski/robots-txt

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

2.1.0 2025-12-11 20:22 UTC

Requires

• php: ^8.4 | ^8.5

Requires (Dev)

• laravel/pint: ^v1.7
• pestphp/pest: ^4.0
• pestphp/pest-plugin-type-coverage: ^4.0
• phpstan/phpstan: ^2.0
• rector/rector: ^2.1.2

MIT

• fkrzski

• seo
• crawler
• robots.txt
• robots
• web-crawler
• php8

Published on Packagist.org on 2025-12-11 20:23 UTC

README

PHP Robots.txt

A modern, fluent PHP package for building valid robots.txt files with type-safe crawlers and fail-fast validation. Invalid paths, sitemap URLs, or crawl delays throw a clear exception as you build — never a broken file at render time.

Requirements

• PHP 8.4 or higher
• A code coverage driver (development only)

Installation

composer require fkrzski/robots-txt

Quick start

Chain rules and render the result with toString():

use Fkrzski\RobotsTxt\RobotsTxt;

$robots = new RobotsTxt();

echo $robots
    ->disallow('/admin')
    ->allow('/public')
    ->crawlDelay(5)
    ->toString();

User-agent: *
Disallow: /admin
Allow: /public
Crawl-delay: 5

Target a specific crawler with userAgent() (or the closure-based forUserAgent()), and write the file to disk with toFile():

use Fkrzski\RobotsTxt\Enums\CrawlerEnum;

(new RobotsTxt())
    ->disallow('/admin')
    ->sitemap('https://example.com/sitemap.xml')
    ->forUserAgent(CrawlerEnum::GOOGLE, function (RobotsTxt $robots): void {
        $robots->disallow('/private')->crawlDelay(10);
    })
    ->toFile(); // writes ./robots.txt

Documentation

Full documentation is hosted at docs.fkrzski.dev/robots-txt:

• Guide — how global and crawler-specific rules compose, output order, and writing to disk.
• API reference — every method with its signature, validation rules, and verified output.
• Crawlers — the full CrawlerEnum list mapped to official user-agent strings.

Contributing

Contributions are welcome — see the Contributing Guide. Run the full test and quality suite with:

composer test

License

Open-sourced software licensed under the MIT License.

Author

PHP Robots.txt was created by Filip Krzyżanowski.