wikimedia/arc-lamp

Log processor for HHVM's Xenon extension

, _(*1)

Arc Lamp

Arc Lamp helps gather stack traces from a running PHP 7 application and publish them in the form of flame graphs and trace logs., _(*2)

See performance.wikimedia.org for a live example., _(*3)

Prerequisites

Client: * Your PHP application, with php-excimer, and php-redis., _(*4)

Processor: * A Redis server. * Python, with python-redis and python-yaml. * The ifne command from moreutils (Debian, Homebrew, source)., _(*5)

Quick start

require_once 'ArcLamp.php';

Wikimedia\ArcLamp::collect( [ 'redis-host' => '127.0.0.1' ] );

To automatically enable this for all web requests and entry points, you can use the PHP auto_prepend_file option. See Wikimedia's own configuration for example., _(*6)

How it works

See also: Profiling PHP in production on Wikimedia Techblog., _(*7)

, _(*8)

The Arc Lamp pipeline comprises of three stages:, _(*9)

1. Capture stack traces.
2. Create trace logs.
3. Generate flame graphs.

Capture stack traces

The php-excimer extension is used to periodically collect a backtrace. It has no run-time overhead by default. When enabled from a web request, it periodically schedules a graceful interrupt at which point it captures a backtrace., _(*10)

These traces can be collected from a PHP callback and dispatched to a socket or file as-needed., _(*11)

The default in ArcLamp.php is to send the trace to a Redis pubsub channel., _(*12)

Arc Lamp was originally created in 2014 for Xenon, a sampling profiler native to the HHVM engine for PHP. To use Arc Lamp with HHVM Xenon, see arc-lamp v1.0 instead., _(*13)

Create trace logs

In Wikimedia's production cluster, Arc Lamp's Redis server resides in the high availability "Tier 1" zone, and thus kept separate from offline performance, research, and statistics services ("Tier 2")., _(*14)

Each of the production web servers uses the ArcLamp::collect client in PHP to send traces to Redis., _(*15)

The arclamp-log service subscribes to the Redis pubsub channel, normalizes the stack traces and write them to the relevant trace log files. The example configuration creates a trace log file for each hour and each day., _(*16)

Within those two time periods, it segregates the trace logs by entry point of the PHP application., _(*17)

For example, the MediaWiki application has index.php, and rest.php (web) and RunJobs.php (CLI) entry points. This results in the following trace logs:, _(*18)

• daily/2019-12-21.all.log
• daily/2019-12-21.index.log
• daily/2019-12-21.rest.log
• daily/2019-12-21.RunJobs.log
• …
• hourly/2019-12-21_20.all.log
• hourly/2019-12-21_20.index.log
• hourly/2019-12-21_20.rest.log
• hourly/2019-12-21_20.RunJobs.log
• …

The arclamp-log service is also responsible for pruning trace logs older than the configured retention period., _(*19)

Generate flame graphs

The arclamp-generate-svgs script runs at a regular interval and creates or updates the flame graph associated with each trace log file. It also maintains a reverse version of each flame graph., _(*20)

For example:, _(*21)

• daily/2019-12-21.all.svgz
• daily/2019-12-21.all.reversed.svgz
• daily/2019-12-21.index.svgz
• daily/2019-12-21.index.reversed.svgz
• …
• hourly/2019-12-21_20.all.svgz
• hourly/2019-12-21_20.all.reversed.svgz
• hourly/2019-12-21_20.index.svgz
• hourly/2019-12-21_20.index.reversed.svgz
• …

The arclamp-generate-svgs script also removes graphs for which a trace log no longer exists., _(*22)

Flamegraphs are generated using Brendan Gregg's flamegraph.pl., _(*23)

Demo

See performance.wikimedia.org for a live example., _(*24)