Operate

Incidents & Diagnostics

How incidents move through their lifecycle, the full set of diagnostic types that can open one, and how Pillar deliberately scopes what host network evidence can and cannot claim.

1.Incident Lifecycle

State	Meaning
DETECTED	A diagnostic condition was observed but has not yet been promoted to a visible incident (e.g. still within a debounce window).
OPEN	An active incident visible in the dashboard and eligible for alert routing.
ACKNOWLEDGED	An operator has claimed the incident; it remains open until the underlying condition clears.
RECOVERING	The underlying condition has improved but not yet cleared for long enough to resolve automatically.
RESOLVED	The condition cleared or an authorized operator marked it resolved. Resolution delivery targets prior successful alert destinations; receipt can still fail.

Severity is one of info, warning, or critical. In practice the incident engine only opens incidents at warning or critical — info-severity findings do not by themselves open an incident.

Investigate and record an outcome

Filter the incident list by state, severity, node, type, or time, then open the incident’s evidence and timeline.
Check freshness and source labels. A likely cause is a diagnostic assessment, not proof of a failed component or client playback behavior.
Operators, Facility Admins, and Organization Admins can acknowledge an incident, add notes, and mark it resolved. Other roles can review evidence.
Record the investigation and verify recovery using fresh host telemetry. Acknowledgment or manual resolution does not fix hardware, stop collection, or prevent future findings.

2.Diagnostic Types

Every incident carries exactly one of these 19 types:

Type	Meaning
disk_latency_outlier	A single drive's latency deviates sharply from its own baseline or its peers.
disk_hardware_degradation	SMART or ATA/SCSI evidence suggests a drive is physically failing.
pool_latency_degradation	Pool-wide latency degradation not attributable to a single outlier drive.
pool_capacity_pressure	A pool is approaching its configured capacity warning or hard threshold.
maintenance_collision	An active or forecast maintenance job overlaps configured operating hours.
rapid_ingest	Capacity growth rate exceeds the configured rapid-growth/rapid-ingest thresholds.
node_offline	A paired node has missed heartbeats past the freshness policy.
network_interface_pressure	Sustained host NIC utilization or a drop burst clears its threshold.
network_interface_error_rate	Host NIC error rate and error count both clear their thresholds.
network_link_instability	A host NIC flaps or drops from up to down, or downshifts link speed.
scrub_slowdown	An active scrub is progressing materially slower than expected.
resilver_collision	An active resilver is forecast to overlap operating hours.
playback_budget_violation	Observed latency exceeds the configured safe storage target for a workload's frame budget.
component_unhealthy	Agent or Observer reports a degraded or disconnected state.
posture_drift	A security posture control that previously passed now fails.
tail_latency_violation	Observer-measured p99/p99.9 latency crosses the tail threshold (requires Observer).
latency_distribution_shift	Observer-measured latency distribution shape shifts materially from baseline.
flush_latency_degradation	Flush (fsync-class) latency degrades, distinct from read/write latency.
queue_vs_device_latency_divergence	Queued time and in-device time diverge, indicating queueing pressure rather than device slowness (requires Observer).

3.Host Network Evidence: What It Answers and What It Never Answers

Question this answers: was the storage host's network interface saturated, dropping packets, or erroring when this happened?

Question this never answers: whether the network path, switch, client, or SAN is healthy. Pillar reads the storage host's own NICs only — it has no visibility past the host's network stack. Language stays host-scoped: Pillar states things like "no host-side NIC saturation was observed," and never states or implies that the network path, switch, client, or SAN is healthy or exonerated.

Network assessments are always capped below stronger hardware and latency findings so they never eclipse a real disk problem — network_interface_pressure and network_interface_error_rate are always warning severity, confidence capped around 0.6 or lower.
The one exception is a clear link failure (an interface that went from up to down, or a flapping storage_uplink interface): that may reach confidence 0.85 and critical severity, and is the only case where a network assessment can set the likely cause to the interface itself.
Every network assessment carries a boundary label (agent_host_nic) and an explicit limits statement that the switch, client, and full network path were not assessed.