gauravharsh / module-log-monitor
A smart, memory-efficient background log monitor and deduplicated error dashboard for Adobe Commerce.
Package info
github.com/gauravharsh15/magento-2-logs-monitor
Type:magento2-module
pkg:composer/gauravharsh/module-log-monitor
Requires
- php: ~8.1.0 || ~8.2.0 || ~8.3.0 || ~8.4.0 || ~8.5.0
- magento/framework: ^103.0.4
- magento/module-backend: *
- magento/module-config: *
- magento/module-cron: *
- magento/module-email: *
- magento/module-store: *
- magento/module-ui: *
Requires (Dev)
None
Suggests
None
Provides
None
Conflicts
None
Replaces
None
README
A professional-grade, high-performance monitoring engine that transforms chaotic var/log files into a clean, actionable, deduplicated error dashboard for Adobe Commerce.
📖 Why This Module?
In most Magento environments, log files are a "dark hole." They grow to gigabytes, are rarely checked until a site crashes, and contain thousands of repetitive lines that bury the actual root cause of issues.
Gaurav_LogMonitor was built to solve this by providing:
- Proactive Health Checks: Identifies "Silent Killers" like failed background data syncs (ERP/CRM) that don't crash the frontend.
- Operational Efficiency: Stops senior developers from wasting hours "log hunting" through massive text files.
- Server Stability: Hard limits on memory, disk reads, run time and table size - when they're hit it records less, rather than loading the server.
- No Double Counting: Each entry's timestamp is checked against the error already listed, so a log that's read again (a daily dated copy, a rotation, a reset) only adds what's new and never turns a known error into a "new" one.
🖥️ Preview & UI
1. Unified Error Dashboard
A consolidated, searchable view of all unique errors across every .log file in your system. This grid groups thousands of identical errors into single, manageable records.
2. Smart Alert Configuration
Easily manage your monitoring settings, define custom email recipients, and set alert thresholds.
✅ Feature Overview
| Feature | Description |
|---|---|
| Byte-offset scanning | Only reads newly-appended bytes on every run, not the whole file. Offsets are 64-bit, so files larger than 4 GB are tracked correctly. |
| Production safety limits | Each run reads at most the newest 16 MB per file and 64 MB in total, stops after 60 seconds, and stores at most 10,000 distinct alerts. Older data beyond the limits is skipped - see Production Safety below. |
| Multi-line trace grouping | Groups a stack trace's continuation lines with the timestamped line that started it into a single error record. |
| Smart deduplication | Errors are grouped per log by their message. With Group Similar Errors on (default), numbers, IDs, UUIDs, hashes and email addresses are ignored, so Order 1001 failed and Order 1002 failed become one alert with a count of 2. |
| Dated files belong to their log | A file named with a date before .log (exception-2026-11-09.log, exception-20261109.log, payment_2026-11-09.log) is recorded under the log it belongs to (exception.log, payment.log). The same error in tomorrow's file is the same alert, not a new one. |
| Timestamp check | Before an entry is counted, its timestamp is compared with the Last Seen of the alert it belongs to. If it isn't newer, it was already counted and is skipped. Re-reading the same lines (a dated copy of a log, a reset, a run cut off before it saved its position) never raises a count or creates a duplicate alert. |
| Rotation detection | Each file is identified by its first line, so a rotated or truncated log is recognised (and read from its start) even if the new file has already grown past the old offset. Rotated copies that don't end in .log (exception.log.1, .gz archives) are never read. |
| Half-written line safety | A last line the logger is still writing is left for the next run instead of being split in two. |
| Severity filtering | Parses channel.LEVEL: from each entry. A configurable minimum severity (default NOTICE) drops noise; unrecognised formats are tagged UNKNOWN and always kept. |
| Log-accurate timestamps | First/last seen come from the log entries themselves (stored in UTC), not from when the scan happened to run. |
| Email once per error | An error is emailed once, when it first crosses the occurrence threshold. After that it only counts up in the admin grid, however often it happens again. Optionally (Email Known Errors Again), an error that keeps recurring is emailed again at most once per re-alert interval. |
| Readable digest emails | Alerts are sorted by severity, marked NEW or RECURRING with "+N since last alert", capped at 50 per email with an "N more" summary. The subject starts with the store's domain (e.g. [staging.example.com]), so production, staging and dev alerts are told apart straight from the inbox; the email also shows the store URL and server hostname. |
| Batched, crash-safe writes | Alerts are written in batches with INSERT ... ON DUPLICATE KEY UPDATE, and the file offset is checkpointed after each batch so an interrupted scan doesn't recount what it saved. |
| Isolated cron group | Jobs run in their own gaurav_logmonitor cron group, in a separate process, so a long first scan never delays other Magento cron jobs. |
| Overlap protection | A non-blocking lock stops the cron job and the CLI command from scanning at the same time. |
| Per-file fault isolation | A corrupt or unreadable log file is skipped and logged - it doesn't abort the scan for every other file that run. |
| Automatic retention cleanup | A daily job purges alerts that haven't been seen within a configurable retention window. |
| Deduplicated admin grid | Filter/sort by severity, log file, occurrence count, email status, first/last seen; drill into the full stack trace per alert; mass delete. |
| Scoped ACL | Menu, dashboard (including the grid's data endpoint) and settings are gated behind their own admin permissions. Mass delete only accepts POST requests with a form key. |
⚙️ Configuration & Navigation
After installation, you can manage the module and view alerts via the following paths:
Admin Dashboard (The Grid)
Navigate to: System > Log Monitor Alerts This is the central command center where you view, filter, and mass-delete deduplicated error logs.
System Configuration
Navigate to: Stores > Configuration > Gaurav Extensions > Log Monitor
All settings are global (Default Config scope): the scanner runs for the whole server, not per website or store view.
General Settings
- Enable Module: Master switch for the background scanner and alert emails.
- Ignore Log Files: Comma-separated file names to skip; wildcards allowed (e.g.
debug.log, payment_*.log). A name also covers its dated files (debug.logalso skipsdebug-2026-11-09.log). Defaults todebug.log, which (when debug logging is on) repeats every entry fromsystem.logandexception.log. On Adobe Commerce, consider addingsupport_report.logtoo if it mirrorssystem.logon your install. - Minimum Severity: Lowest level recorded (default Notice and above).
UNKNOWNentries are always kept. - Group Similar Errors: Ignore numbers/IDs/hashes/emails when grouping (default Yes). Changing it starts new alert rows for errors already recorded.
- Alert Retention (Days): How long an alert is kept after it was last seen before the daily cleanup job deletes it (default 30).
0disables cleanup entirely.
Email Alert Settings
- Recipient Emails: Comma-separated list of developers to receive alert emails. If blank, no email is sent (there is deliberately no fallback to the store contact address, which is often a customer-service inbox). Invalid addresses are skipped.
- Minimum Occurrences Threshold: How many times an error must occur (since it was first seen, or since it was last emailed) before an alert email is sent.
- Email Known Errors Again: No by default: each error is emailed once, when it first appears, and afterwards only its count goes up in the grid. Yes: an error that keeps happening is emailed again, at most once per Re-alert Interval.
- Re-alert Interval (Hours): Shown when Email Known Errors Again is Yes. Minimum time between two emails about the same error (default 24).
0includes a recurring error in every hourly email. New errors are never delayed.
🏗️ Business Flow
1. Scanning - gaurav_logmonitor_process cron (every 5 minutes) / bin/magento logmonitor:process
- Skip entirely if the module is disabled.
- Acquire a non-blocking lock; if a previous scan is still running, skip this run rather than racing it.
- List
var/log. For each*.logfile not on the ignore list:- Load its last-read byte offset and first-line fingerprint from the state table.
- If the first line changed, the file was rotated: it's a new file under the same name. Rotated copies themselves are only read if their name still ends in
.log, as a file of their own. - If the file was rotated or shrank (or a reset was requested), start from byte 0. A file never seen before also starts from byte 0.
- If more than 16 MB is waiting (first scan of an old log, an error storm, cron having been down), skip ahead and read only the newest 16 MB.
- Seek straight to the offset and stream lines with
fgets()in 64 KB pieces - no full-file or full-line load. - A line starting with a Magento timestamp (
[2026-10-20T10:15:00.123456+00:00]or[2026-10-20 10:15:00]) begins a new entry; every following non-timestamped line (stack trace) is appended to it, capped at 500 lines / 64 KB. - A half-written last line is left for the next run.
- Entries below the minimum severity are dropped. The rest are fingerprinted (message without timestamp, normalised if grouping is on, MD5).
- Each entry's timestamp is compared with the Last Seen of its alert (looked up once per alert per file). Not newer: already counted, skipped. Newer, or no such alert: collected in memory.
- Every 100 unique errors (or 512 KB of trace text) the batch is written to the DB; the offset is checkpointed every 4 MB.
- Once the run has read 64 MB or run for 60 seconds, it stops between two entries; the rest waits for the next run.
- A failure on one file is logged and the loop continues with the next file.
- Release the lock.
2. Deduplication - inside every batch write
- Rows are matched on
(log_file, error_hash).log_fileis the log a file belongs to, soexception-2026-11-09.logmatches theexception.logrows. - Exists:
occurrence_countis increased by the number of new occurrences;first_seen_at/last_seen_atwiden to include the new entries. The stored trace isn't rewritten. - New: a row is created with severity, a 250-byte preview (timestamp stripped), the full entry (capped at 20,000 bytes), and the entry's own timestamps - unless 10,000 alerts are already stored, in which case it's dropped.
3. Alerting - gaurav_logmonitor_send_alerts cron (hourly)
- Skip if disabled or there's no valid recipient (alerts stay pending).
- Select every alert where
occurrence_count >= last_notified_count + thresholdthat has never been emailed. With Email Known Errors Again on, alerts emailed before are included too, once they haven't been emailed within the re-alert interval. - Count them in SQL and load only the top 50 (by severity, then occurrences since the last email); the rest appear as an "N more" line. A banner warns if the 10,000-alert limit has been reached.
- Send to the configured recipients. The subject leads with the default store view's domain, e.g.
🚨 [staging.example.com] Log Monitor: 3 error(s) need attention. - On success, mark all selected alerts notified (in chunks of 1,000 IDs), snapshotting their counts and the time sent.
- On failure, log the error; the alerts stay pending and are retried next hour.
4. Cleanup - gaurav_logmonitor_cleanup cron (daily, 2:30am)
- If retention is enabled (> 0 days, default 30), deletes any alert whose
last_seen_atis older than the configured window. - Forgets the read position of log files that no longer exist.
5. Investigation - Admin Dashboard
- System > Log Monitor Alerts lists every deduplicated alert: ID, severity, log file, preview, occurrence count, email status, first seen, last seen (and an optional Last Emailed column) - all filterable and sortable, most recently seen first.
- View Trace opens the full entry with its details for a single alert.
- Mass Delete clears out selected alerts (there is currently no separate "resolve/mute" state - deleting is how an alert is cleared from the grid; if the same error recurs afterward it will simply be recreated as new).
🛡️ Production Safety
The module is built to lose data rather than load the server. Every limit below is a constant in Model/LogReader.php.
| Limit | Value | When it's hit |
|---|---|---|
| New data read per file, per run | newest 16 MB | Older unread data is skipped for good (first scan of an old log, error storms, cron downtime). |
| Data read per run, all files | 64 MB | The run stops between two entries; the rest is read next run. |
| Run time | 60 s | Same as above. |
| Distinct alerts stored | 10,000 | New kinds of errors are dropped; existing alerts keep counting. The alert email shows a warning. |
| Alerts per email | 50 | The rest are summarised as "N more"; at most one email per hour. |
Measured on a 40 MB log: 16 MB read in under 0.2 s, with under 0.1 MB of extra memory. The alert email job uses well under 1 MB extra with thousands of pending alerts. All jobs run in their own cron process, so a failure in this module can't stop other Magento cron jobs.
Kill switch: bin/magento config:set logmonitor/general/enabled 0 && bin/magento cache:clean config stops scanning and emails at the next cron run.
📋 Requirements & Compatibility
| Supported | |
|---|---|
| Magento Open Source / Adobe Commerce | 2.4.4 - 2.4.9 |
| PHP | 8.1, 8.2, 8.3, 8.4, 8.5 (whichever your Magento version supports) |
| Database | MySQL 8.0 / 8.4, MariaDB 10.4+ |
| Logger | Monolog 1, 2 and 3 line formats (Magento's default var/log handlers) |
The floor of 2.4.4 comes from the PHP 8.1 minimum; the code itself uses nothing newer than Magento 2.3 APIs (declarative schema, HTTP-method action interfaces, the lock manager). The XML config is validated, and the scanner and SQL are tested, against the 2.4.7 (Monolog 2) and 2.4.8 (Monolog 3) framework code.
Limitations
- Only
var/logon the server that runs cron is scanned. On a multi-node setup, errors written to other nodes' localvar/logaren't seen - ship logs to shared storage or a central log service for those. - Subdirectories of
var/logaren't scanned. - Entries written between the last scan and a rotation (at most one scan interval, 5 minutes) are missed when the rotated copy is compressed or renamed to a name that doesn't end in
.log. - Last Seen is stored in whole seconds. If the same error happens more than once within one second and a scan falls in between, the later occurrences in that second are taken as already counted. Counts can come out slightly low for very frequent errors, never high.
- An alert deleted from the grid comes back (as new) if lines containing it are read again, e.g. from a dated copy of the log or with
--reset. - Counts are lower than reality whenever a safety limit skips data (e.g. during an error storm) - the alert still appears, but its count is a sample.
--resetre-reads every file (up to its newest 16 MB). Entries already counted are skipped, so it only re-creates alerts that are missing from the list (deleted ones, or new groupings after changing Group Similar Errors).
🛠️ Installation & Setup
composer require gauravharsh/module-log-monitor bin/magento module:enable Gaurav_LogMonitor bin/magento setup:upgrade bin/magento setup:di:compile bin/magento cache:flush
Or copy the module to app/code/Gaurav/LogMonitor and run the same commands from module:enable onwards.
The jobs run in their own cron group. If your crontab runs bin/magento cron:run with a --group filter, add --group=gaurav_logmonitor (a plain bin/magento cron:run covers it automatically).
Upgrading from 1.2.x
- Errors are now emailed once by default. To keep the previous behaviour (recurring errors emailed again, at most once per Re-alert Interval), set Email Known Errors Again to Yes.
- The Recover Entries from Rotated Logs setting is gone: rotated and compressed copies are no longer opened. No database changes.
- Alerts that earlier versions recorded under a dated file name (e.g.
exception-20261109.log) stay as they are, and new occurrences count on theexception.logalert. Filter the grid by Log File and delete the dated ones, or let retention remove them.
Upgrading from 1.1.x
setup:upgradeconverts the stored byte offsets to 64-bit, adds the rotation fingerprint and last-emailed columns, stopslast_seen_atfrom changing when an email is sent, and drops the unusedgaurav_logmonitor_archive_statetable.- Existing offsets are kept - logs are not rescanned.
- With Group Similar Errors on, an error whose message contains numbers/IDs gets a new hash, so it starts a new alert row; the old row stops growing and ages out through retention (or delete it). Set Group Similar Errors to No to keep the previous exact-match behaviour and hashes.
debug.logis now ignored by default unless you've saved your own Ignore Log Files value.