Storage monitoring for TrueNAS post-production facilities

Without the optional Observer, latency is the average wait per collection interval, not per-I/O percentiles. A short spike inside an interval can be averaged away. When data is unavailable, Pillar shows it as missing rather than guessing.

This is a storage contention investigation prompted by a hypothetical playback report, reproduced on a disposable TrueNAS lab system with synthetic media. No editing application was tested. Each step is labeled as an observation, an inference or an action.

1. 1. Symptom Observation

   An editor reports that playback stalled at about 14:20. The operator opens that system's timeline for 14:00–14:40.

2. 2. Evidence Observation

   Pillar shows what the Agent recorded around that time: whether a scrub was scanning the pool, its progress, and the per-disk throughput and average wait in the same intervals.

3. 3. Investigation Inference

   If the scrub overlapped the report and disk wait rose with it, storage contention becomes a reasonable hypothesis. It is not proof: the network, the client and the application are still candidates, and a single report may be coincidence.

4. 4. Decision Action

   The operator gets approval to move the scrub schedule in TrueNAS outside edit hours, and asks editors to note the times of any further stalls. Pillar does not change the schedule.

5. 5. Follow-up Observation

   After the change, compare another observation window. Report an improvement only if the stall reports and the measurements actually change.

What the lab run measured

The run has a baseline window, a window with a scrub running, and a recovery window. Three recorders run the whole time and are kept separate: a fixed-rate reader standing in for one 24 fps stream (4 MiB per frame), TrueNAS's own zpool iostat, and the Pillar Agent.

Prerequisites

• An administrator account with shell access (System Settings → Shell, or SSH).
• The TrueNAS Apps service running, so Docker is available.
• Outbound HTTPS to Pillar (proxies are supported). No inbound ports.
• Approval from whoever owns the system to run a third-party container on it, and a maintenance window for the install.

Still being verified on TrueNAS

Staying running across reboots, updates and rollback, uninstall, and behavior across TrueNAS upgrades have not yet been demonstrated on TrueNAS. We will not describe them as supported until they have.

Agent and Observer

Deployment guide

We haven't measured an install time on TrueNAS yet, so we don't quote one. Here is the work involved. Useful history, such as capacity runway, builds up over days after that.

1. Confirm compatibility and approval. Check your release and hardware against the table above, and get sign-off to run a container on the system.
2. Run and review the preflight. Download the read-only preflight script, read it, then run it. It reports the OS, architecture, block-device access and outbound connectivity.
3. Generate a pairing command and install. A Facility or Organization Admin creates a one-time command in Settings → Agent pairing and runs it in the TrueNAS shell. The installer verifies the signed release and image before starting the Agent.
4. Verify. Check that the system appears with a fresh heartbeat, and that the expected pools and drives and the available telemetry are all present.
5. Configure facility context. Set the timezone, operating hours and alert routing before relying on notifications.

Access the Agent is given

• Read-only root filesystem, all capabilities dropped except SYS_RAWIO, no-new-privileges, process and memory limits

  Limits what the container can change.

  The container image cannot be modified at runtime. A credential volume holds its API key.

• Read access to every block device

  Needed for SMART and device inspection.

  The process can technically read raw disk blocks. Pillar reads metadata, not file contents, and does not mount your datasets. The permission alone does not prevent block reads.

• Access to /dev/zfs

  Needed to read pool and scrub state.

  Without CAP_SYS_ADMIN, the container can read pool state but cannot pause, resume or change pools.

• Read-only /proc/diskstats, /sys/block and host version files

  Needed for disk counters and platform detection.

  No Docker socket, no host shell.

What leaves the system

Outbound HTTPS only (TLS 1.3 for the Agent), with no listening ports. The Agent sends hostname, machine ID, OS, kernel and TrueNAS version, disk models, serials and SMART data, block and network counters, pool and dataset names and capacity, and a runtime posture report. It does not send file contents or file names. An optional setting partly redacts disk serials and MAC addresses; hostnames, pool names and dataset names are not anonymized. See Security and Data, retention & support.

Actions and removal

On TrueNAS, Pillar is observation only: it does not pause scrubs, blink LEDs or run commands on the system. Removal is pillar uninstall agent --yes --purge, followed by removing the system in the dashboard. Cancelling a subscription does not uninstall the Agent. Uninstall has not yet been run on TrueNAS, so plan to check it during evaluation.

Deployment guide

Pricing is flat per plan, with no per-terabyte metering or per-seat charges. An appliance is one storage system running the Agent; the Observer on that same system is not billed separately. Studio Fleet includes 90 days of telemetry history. The trial lasts 14 days, covers 1 appliance and needs no card. It starts when the first system connects, not at sign-up, so waiting for approval doesn't use up trial days. To stop renewal, cancel in the billing portal at least 30 days before the end of your current subscription period. See Plans & billing.

An evaluation checklist

Decide what success looks like before you install. A good evaluation ends in a clear decision, even if the decision is that Pillar doesn't fit.

• The intended system connects, and the expected pools, drives and telemetry are present.
• A real or staged incident can be investigated from the shared timeline.
• Scrub or resilver overlap with your operating hours is visible, where your system runs them.
• Capacity information is useful once enough history has built up, and its uncertainty is clear.
• The installation, permissions and day-to-day running are acceptable to your facility.