Incident Attribution & Metric Interpretation

Agent vs. Observer: What Each Can Measure

Pillar Agent alone reads /proc/diskstats and reports mean read/write await, interval latency deviation, queue pressure, and utilization. True per-I/O percentiles, request-size distribution, and the queued-vs-device time split require the optional Pillar Observer (eBPF) component. See Agent & Observer reference for install and capability details.

I/O Latency Distribution (Observer)

With Pillar Observer, the Storage Playback tab shows p50, p95, p99 and p99.9 for reads, writes and flushes over 1 hour to 7 days, by pool, VDEV or drive. Every value is recomputed from per-I/O histogram bucket counts: Pillar merges bucket counts across time and never averages stored percentiles. A pool or VDEV line shows the worst member drive for each interval (for example, “Pool p99 shown as worst member-drive p99”), because averaging drives would hide the one that stalls.

A percentile is shown only when enough I/Os back it (100 samples for p99 and 1,000 for p99.9 by default, configurable in Settings); otherwise Pillar shows “insufficient samples”. The observed window duration and sample count are always displayed. Without Observer, Pillar keeps the Agent’s mean read latency view and states that true latency percentiles require Pillar Observer.

Tail-latency diagnostics look at several recent windows together, a drive’s own percentile baseline, peer drives, queue pressure, SMART and maintenance state, so a single slow bucket never marks a drive unhealthy. History is kept as raw 10-second windows for 7 days, 1-minute rollups for 30 days, 15-minute rollups for up to 90 days and 1-hour rollups for up to 1 year, limited by your plan’s telemetry retention.

Each Observer measurement window (observer_windows) carries its own provenance, support state, and quality counters, and the heartbeat response acknowledges every window Pillar Cloud stored (observerWindows.acks) so the Agent can retry only what wasn’t confirmed. Pillar Cloud only ever plots qualified windows — ones the Agent, Observer, and Cloud all agree are a certified measurement. A window (or one drive’s series within it) is excluded, and never plotted, when it is a synthetic/demo sample outside demo mode, missing its eBPF probe digest, not currently “measuring”, affected by dropped or lost events, or reports a data-quality flag such as a reader backlog or a clock step. Excluded windows are counted, not discarded — the chart caption reads “N windows excluded: reason (count), …” so a gap in the trend is never silently unexplained.

A percentile whose rank falls in the open, unbounded top bucket (over 256 ms) is a lower bound, not an exact value — Pillar never invents a number past the last real bucket boundary. These render as “≥256 ms” with a tooltip explaining the value is a floor, not an average or an estimate: the true latency could be higher. A lower bound still proves a safe-target breach whenever it already exceeds the target. Chart captions also report Observer’s own connection status (sampler, probe digest prefix, session, and age of the last window) and, for the most recent hour of raw windows, sequence-gap coverage (“Coverage X% · N windows missing (sequence gaps)”) so a dropped connection is visible even when the chart still shows data either side of the gap.

Deferred (tracked, not yet built): request-size distribution charts (the series is stored but has no chart yet), a queued-vs-device (“total”) latency view, latency data export/CSV, fleet-level latency quality badges, and richer simulator scenarios for lossy/sparse Observer delivery.

Verify real-time host await latencies directly from the terminal:

iostat -xz 1 5