Remediation Runbooks

Advisory Remediation Runbooks

Actionable, copy-paste terminal workflows for Systems Engineers and Technical Directors during active post-production incidents.

Runbook 1: Resolving Active Scrub Collisions

INCIDENT: ZFS scrub running during critical editorial review causes playback stutter and dropped frames.

Option A: Immediate Scrub Pause (Fastest Recovery)

Halts scrub disk I/O immediately, freeing 100% of spindle queue depth for video streams:

bash

sudo zpool scrub -p tank_media

Option B: Dynamic Priority Throttling (Keep Scrub Running at Low Priority)

Injects kernel delays into scrub I/O chunks so media playback passes unimpeded:

bash

# Throttle scrub background priority to prioritize edit suites
echo 8 | sudo tee /sys/module/zfs/parameters/zfs_scrub_delay
echo 1 | sudo tee /sys/module/zfs/parameters/zfs_vdev_scrub_max_active

Resume Scrub During Maintenance Window

bash

# Restore full scrub speed and resume
echo 0 | sudo tee /sys/module/zfs/parameters/zfs_scrub_delay
echo 3 | sudo tee /sys/module/zfs/parameters/zfs_vdev_scrub_max_active
sudo zpool scrub tank_media

Runbook 2: Isolating and Replacing a Failing Physical Drive

INCIDENT: Pillar alerts that drive /dev/sdd has climbing reallocated sectors and p99 await latency exceeding 100ms.

Step 1: Check SMART Error Counters

bash

sudo smartctl -A -d sat /dev/sdd | grep -E "Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable"

Step 2: Flash Physical Chassis Drive Bay LED

Activates the enclosure locate LED so technicians pull the correct drive sled:

bash

# Illuminate physical chassis drive bay LED
sudo ledctl locate=/dev/sdd

Step 3: Offline Drive, Swap Hardware, and Trigger Online Resilver

bash

# 1. Identify persistent drive serial
DISK_ID=$(udevadm info --query=property --name=/dev/sdd | grep -E "ID_SERIAL=" | cut -d'=' -f2)

# 2. Offline failing drive in ZFS pool
sudo zpool offline tank_media /dev/disk/by-id/ata-${DISK_ID}

# 3. Physically swap drive sled, then replace & initiate resilver
sudo zpool replace tank_media /dev/disk/by-id/ata-${DISK_ID} /dev/disk/by-id/<NEW_DRIVE_ID>

Runbook 3: Diagnosing & Testing Standby Sleep Preservation

VERIFICATION: Assert that cold archive drives in low-power standby are never woken up by the monitoring daemon.

Step 1: Query Standby Status Non-Destructively

bash

# Query power status without waking platters (Exit code 2 = Standby)
sudo smartctl -i -n standby /dev/sdb
echo "Exit status: $?"

Step 2: Force Test Drive into Standby and Inspect Container Logs

bash

# Force drive into standby sleep for test
sudo hdparm -y /dev/sdb

# Verify agent log confirms sleep state preserved
docker logs pillar-agent 2>&1 | grep "Device is in STANDBY"