friends-of-hyva / magento2-crawler-session
Prevent crawlers from creating a session
Installs: 1 705
Dependents: 0
Suggesters: 0
Security: 0
Stars: 11
Watchers: 2
Forks: 2
Open Issues: 0
Type:magento2-module
Requires
- php: ^8.1
- jaybizzle/crawler-detect: ^1.2
- magento/framework: *
README
Prevent crawlers from creating a session!
It's insane how many requests are coming from crawlers & bots nowadays. Every request creates an unnecessary session. If you block the SEO tools via robots.txt there are still plenty of search engines left. 100 requests per minute is not unusal. This leads to a lot of sessions which could affect some limits like Redis "max_concurrency".
This module prevents the initiation of a session if a crawler is detected. The detection is based on:
https://github.com/JayBizzle/Crawler-Detect
Installation
Install the package via composer:
composer require friends-of-hyva/magento2-crawler-session php bin/magento setup:upgrade
Usage
After installation, you need to enable the module:
Stores > Configuration > General > Web > Prevent Crawler Session
Configuration
Enabled
If enabled, the detected crawler will no longer create a session.
Additional Blacklist
If the provided list from crawlerdetect.io is not enough, you can define your own custom user agents here.
Whitelist
For situations where you need to allow a user agent that is blacklisted, you can do so here.
Log
If enabled all blocked user agents are logged to var/log/crawler-session.log
.
But this is only intended for temporary debugging purposes. The filesize can get big very fast!