wikimedia / arc-lamp
Flame graphs and log processing for PHP stack traces.
Installs: 3 032
Dependents: 0
Suggesters: 0
Security: 0
Stars: 44
Watchers: 16
Forks: 5
Language:Shell
Requires
- php: >=7.2.0
- ext-redis: *
Requires (Dev)
This package is auto-updated.
Last update: 2024-11-08 08:06:37 UTC
README
Arc Lamp
Arc Lamp helps gather stack traces from a running PHP 7 application and publish them in the form of flame graphs and trace logs.
See performance.wikimedia.org for a live example.
Prerequisites
Client:
- Your PHP application, with php-excimer, and php-redis.
Processor:
- A Redis server.
- Python, with
python-redis
andpython-yaml
. - The
ifne
command frommoreutils
(Debian, Homebrew, source).
Quick start
require_once 'ArcLamp.php'; Wikimedia\ArcLamp::collect( [ 'redis-host' => '127.0.0.1' ] );
To automatically enable this for all web requests and entry points, you can use the PHP
auto_prepend_file
option. See Wikimedia's own configuration for example.
How it works
See also: Profiling PHP in production on Wikimedia Techblog.
The Arc Lamp pipeline comprises of three stages:
- Capture stack traces.
- Create trace logs.
- Generate flame graphs.
Capture stack traces
The php-excimer extension is used to periodically collect a backtrace. It has no run-time overhead by default. When enabled from a web request, it periodically schedules a graceful interrupt at which point it captures a backtrace.
These traces can be collected from a PHP callback and dispatched to a socket or file as-needed.
The default in ArcLamp.php
is to send the trace to a Redis pubsub channel.
Arc Lamp was originally created in 2014 for Xenon, a sampling profiler native to the HHVM engine for PHP. To use Arc Lamp with HHVM Xenon, see arc-lamp v1.0 instead.
Create trace logs
In Wikimedia's production cluster, Arc Lamp's Redis server resides in the high availability "Tier 1" zone, and thus kept separate from offline performance, research, and statistics services ("Tier 2").
Each of the production web servers uses the ArcLamp::collect
client in PHP to send traces to Redis.
The arclamp-log
service subscribes to the Redis pubsub channel, normalizes the stack traces and
write them to the relevant trace log files. The example configuration creates
a trace log file for each hour and each day.
Within those two time periods, it segregates the trace logs by entry point of the PHP application.
For example, the MediaWiki application has index.php
, and rest.php
(web) and RunJobs.php
(CLI)
entry points. This results in the following trace logs:
daily/2019-12-21.all.log
daily/2019-12-21.index.log
daily/2019-12-21.rest.log
daily/2019-12-21.RunJobs.log
- …
hourly/2019-12-21_20.all.log
hourly/2019-12-21_20.index.log
hourly/2019-12-21_20.rest.log
hourly/2019-12-21_20.RunJobs.log
- …
The arclamp-log
service is also responsible for pruning trace logs older than the configured
retention period.
Generate flame graphs
The arclamp-generate-svgs
script runs at a regular interval and creates or updates the flame graph
associated with each trace log file. It also maintains a reverse version of each flame graph.
For example:
daily/2019-12-21.all.svgz
daily/2019-12-21.all.reversed.svgz
daily/2019-12-21.index.svgz
daily/2019-12-21.index.reversed.svgz
- …
hourly/2019-12-21_20.all.svgz
hourly/2019-12-21_20.all.reversed.svgz
hourly/2019-12-21_20.index.svgz
hourly/2019-12-21_20.index.reversed.svgz
- …
The arclamp-generate-svgs
script also removes graphs for which a trace log no longer exists.
Flamegraphs are generated using Brendan Gregg's flamegraph.pl.
Demo
See performance.wikimedia.org for a live example.