Agent & Observer Reference

• --preflightRuns the non-destructive hardware/permission readiness suite and exits with a deterministic POSIX code (see Troubleshooting).
• --jsonEmits preflight results as a single JSON document instead of human-readable text.
• --snapshotCollects a telemetry snapshot as JSON to stdout and exits; review and redact operational identifiers before sharing.
• --versionPrints the Agent's semantic version and exits.

Pillar is observe-only: the Agent has no code path that changes the host — no scrub pause/resume, no LED or other sysfs writes, no remote command execution of any kind. For wire compatibility, every heartbeat still reports capabilities.agent.actions as an always-empty array; Cloud continues to accept older Agents that report a non-empty list, but never dispatches anything to them.

Each vdev in a pool's vdevs array may additionally report read_errors, write_errors, checksum_errors (non-negative integer counts since the last zpool clear/import) and a members array (per-leaf-disk state and the same three counters, in zpool status / mdstat order). All four fields are optional, ZFS-only today, and omitted where the platform cannot report them (mdadm, Unraid). Every members[].disk_id also appears in that vdev's disk_ids. A cloud must be upgraded before it will recognize the new raid4/raid5/raid6/raid10/reshape enum values an updated Agent may start sending.

A pool's capacity always carries total_bytes. used_bytes, free_bytes and allocated_pct are omitted when the Agent cannot measure filesystem usage — today, the generic Linux platform (md RAID or plain ext4/XFS) when the Agent container has no access to the host filesystems. ZFS, TrueNAS and Unraid pools send all fields. Cloud treats an omitted field as unknown, never as 0: the dashboard shows the pool's allocation as Unknown, produces no capacity forecast for it, and excludes it from facility totals. Older Agents that always send all three remain accepted.

{"observer_version":"0.3.0","window_ms":10000,"emitted_at":"2026-09-12T15:03:14Z",
 "subprotocol":2,"sampler":"ebpf","probe_set_digest":"004d0568...","session_id":"d3cbc8c43aec9ac9b85d923eaadc1bbb",
 "sequence":42,"window_start_at":"2026-09-12T15:03:04.000Z","window_end_at":"2026-09-12T15:03:14.000Z",
 "duration_ms":10000,"latency_layout":"log2-1ms-10","size_layout":"log2-4kib-9",
 "percentile_algorithm":"bucket-bounds-v2",
 "support":{"state":"measuring","reason_codes":[],"required_hooks":"attached",
   "insertion_hook":"attached","size_source":"kernel_issue"},
 "quality":{"events_received":250,"events_lost":0,"loss_scope":"none","reason_codes":[], "...":"..."},
 "latency_series":[
   {"disk_id":"wwn-0x1","operation":"read","timing_basis":"service","samples":100,
    "buckets":[10,20,30,25,10,3,1,1,0,0],"error_count":1,"worst_bucket_index":7,
    "tail_threshold_ms":20,"tail_violation_count":5,"requeue_count":6,
    "p99":{"lower_ms":32,"upper_ms":64,"estimate_ms":64.0,"censored":false,"eligible":true}, "...":"...",
    "split_coverage_count":80,"split_coverage_ratio":0.8,
    "quality":{"loss_scope":"none","reason_codes":[]}},
   {"disk_id":"wwn-0x1","operation":"read","timing_basis":"queued","samples":80,
    "buckets":[40,25,10,5,0,0,0,0,0,0], "...":"..."}],
 "request_size_series":[
   {"disk_id":"wwn-0x1","operation":"read","source":"kernel_issue","population":100,"samples":100,
    "buckets":[20,15,10,20,15,10,5,3,2],"layout":"log2-4kib-9"}],
 "security":{"checked_at":"2026-09-12T15:00:00Z","capabilities_effective":["CAP_PERFMON","CAP_BPF"],
   "extra_capabilities":[],"tcp_listeners":0,"read_only_root":true,"network_isolated":true,
   "dev_mode":false}}

Each latency_series entry is one (disk, operation, timing_basis) distribution: service covers every completed request; queued and total are built only from requests whose kernel insertion (block_rq_insert) was also observed, and the service entry's split_coverage_count/split_coverage_ratio say how many of its samples had that split. The frame carries no v1 latency_histograms/request_size_histograms arrays; the Agent derives capabilities.observer.block_latency_histogram, request_size_histogram, and queue_service_split (a queued or total series is present) from these series. An idle window sends latency_series/request_size_series as [], never omitted. The Observer uses bounded in-memory delivery and does not persist windows to disk; undeliverable windows can be dropped and are counted in the next window's quality.prior_ipc_delivery_drops. The Agent separately spools accepted windows when heartbeat delivery fails. Entries marked "..." are abbreviated; Cloud recomputes percentiles from bucket counts rather than trusting the wire bounds.

A v2-capable Agent forwards each Observer measurement window as one observer_windows[] entry — identity (session_id, monotonic sequence), provenance (sampler, probe-set digest, Observer version), source timestamps, Observer’s own support state and quality counters, and the per-disk latency/request-size series (bucket counts and sample counts only — Cloud always recomputes percentiles from counts, never trusts a wire percentile). Every heartbeat also carries one observer_status summary — sampler, probe digest, session, last window sequence/age, whether Observer is currently “measuring”, sequence-gap counts, and the Agent’s own durable delivery-queue depth — so the dashboard can distinguish “Observer not installed” from “installed but not connected” from “connected but not measuring” from “measuring”, independent of whether any new window happened to arrive this heartbeat.

Cloud stores every window it can parse (even a status-only or excluded one, for audit) and acknowledges each by identity in the heartbeat response — observerWindows.acks[], one of stored, duplicate (identical redelivery), or rejected (a reason such as invalid_window, identity_conflict, unsupported_subprotocol, or unsupported_layout) — plus an unidentified count for windows whose identity couldn’t even be parsed. The Agent’s own durable queue removes a window once it sees a stored/duplicate/rejected ack, retries an unacknowledged window on the next heartbeat, and evicts it (counted) after 10 unacknowledged attempts against a responding Cloud. A pre-v2 Observer frame (subprotocol below 2) is reported as status only.

Storing a window is not the same as it being qualified. Cloud independently judges each window — and each drive’s series within it — authoritative or not, and only an authoritative, current-basis window is ever plotted; the rest are counted into an “excluded” total by reason (synthetic/demo sampler outside demo mode, a missing probe digest, not currently measuring, an unsupported layout or percentile algorithm, lost/dropped/malformed events, a reader backlog, or a clock step). A percentile whose rank falls in the open, unbounded top histogram bucket is reported as a lower bound (rendered “≥256 ms”), never as a fabricated exact value.

Agent, Observer and CLI are released together under one version, and hosts check for updates once a day. Pillar Cloud supports the latest release and keeps accepting the previous release for 72 hours after it first sees a new one, so the daily update has time to arrive. After that, an older Agent’s heartbeats get HTTP 426 Upgrade Required (with X-Pillar-Min-Version and X-Pillar-Upgrade-Url); the Agent keeps running and retries with backoff until it is updated. Stable releases are never accepted below 1.0.0.

Each system’s Overview shows its release status: Current, Updating (behind the latest release, with the time it stops being accepted) or Unsupported (below the minimum). If a host stays behind, run sudo pillar update all and check that unattended updates are enabled (pillar config agent get).