Skip to documentation
PILLARDocumentation
Operate

Investigation runbooks

Start with current evidence, confirm the target, and verify the outcome. Commands below are examples for an authorized host administrator, not actions Pillar runs automatically.

Before a host change

Confirm the appliance version, pool, device identity, redundancy, and recovery plan. Follow the vendor’s supported procedure. Device names such as /dev/sdd can change after reboot; match persistent identifiers and serial numbers. Do not offline, replace, or force a production drive into standby based on one alert.

1.Investigate scrub contention

If elevated latency overlaps a scrub, compare the incident time with the host’s maintenance state and foreground I/O. Correlation is evidence to investigate, not proof that the scrub caused a playback problem.

bash
# Inspect the intended pool; substitute your verified pool name
zpool status "YOUR_POOL"
zpool iostat -v "YOUR_POOL" 1 5

When your platform supports pausing an active scrub and the storage owner approves the delay, use your host UI or the following commands — Pillar is observe-only and has no action of its own to run this for you. A resilver is a different operation; do not apply scrub instructions to it.

bash
# Pause an active scrub, then check reported state
sudo zpool scrub -p "YOUR_POOL"
zpool status "YOUR_POOL"

# During the agreed maintenance window, resume and verify
sudo zpool scrub "YOUR_POOL"
zpool status "YOUR_POOL"

Recheck latency and the incident after the change, and record when integrity checking will resume. Pausing may reduce competing I/O; it does not guarantee immediate recovery. Kernel tuning parameters vary by ZFS version and appliance support policy.

Command semantics: OpenZFS scrub reference. Check the manual for your installed version.

2.Investigate a suspected drive problem

  1. Open the system’s Drives and Topology tabs. Compare SMART findings, latency history, maintenance state, and mapping confidence.
  2. Confirm the persistent device identifier and serial on the host. Query health using the device type required by your controller.
  3. Use locator LEDs only where supported and independently confirm the physical bay. A light alone is not sufficient authorization to remove a drive.
  4. Before replacement, verify backups, pool redundancy, other degraded members, and the vendor’s replacement procedure. Additional tests or a resilver can impose load on an already degraded pool.
bash
# Read identity and health; replace with a verified persistent device path
sudo smartctl -i -H -A /dev/disk/by-id/YOUR_VERIFIED_DEVICE_ID
zpool status "YOUR_POOL"

SMART commands can wake a disk depending on transport and options. Do not start extended self-tests during production without reviewing their impact. After any approved repair, verify pool health and fresh telemetry before resolving the incident.

3.Check standby behavior

The Agent requests SMART checks that avoid intentionally waking standby drives. Results depend on the drive, controller, bridge, and other host processes. Cached health information for a sleeping disk is not a fresh hardware assessment.

bash
# On a suitable ATA device, query without intentionally waking standby media
sudo smartctl -n standby -i /dev/disk/by-id/YOUR_VERIFIED_DEVICE_ID
# Read both the output and exit status; errors can have multiple causes
echo "Exit status: $?"

Compare Agent logs with the host’s power-state reporting. Validate behavior on a spare device in a maintenance window before changing production sleep policy. Avoid forced standby tests on active pool members.

Record the outcome

Add the evidence, approved action, and verification result to the incident. See Maintenance for how Pillar verifies host changes from subsequent telemetry.
← Project chargebackTroubleshooting →