Diagnostics & Operations

Troubleshooting & Agent Lifecycle

Detailed procedures for diagnosing TLS handshake failures, verifying SAS expander raw I/O, consulting POSIX exit codes, and performing graceful de-provisioning.

1.Diagnosing TLS Handshake Failures

The Agent requires TLS 1.3 or newer and validates the server certificate against its bundled root store. An intercepting egress proxy that decrypts outbound port 443 traffic with a private internal CA certificate will break that validation.

Replace YOUR_PILLAR_DOMAIN with the hostname in your Agent configuration. Run this probe from the storage host to inspect the certificate issuer:

bash

curl --fail --show-error --verbose --tlsv1.3 --tls-max 1.3 https://YOUR_PILLAR_DOMAIN/api/v1/status

If your organization's policy requires the interception to remain in place, give the Pillar CLI the approved CA bundle (PEM). It installs it as /etc/pillar/custom-ca.pem and keeps the Agent trusting it across every repair and update; certificate validation stays enabled. For a new install, set PILLAR_CUSTOM_CA_FILE=/path/to/approved-ca.pem in the hosted install command's environment (curl ... | sudo PILLAR_CUSTOM_CA_FILE=... sh). On a host that is already installed, run:

bash

sudo pillar install agent --repair --custom-ca-file /path/to/approved-ca.pem

If your network requires an explicit forward proxy, configure it via the standard proxy environment variables:

bash

docker run -d \
  --name pillar-agent \
  ... \
  -e HTTPS_PROXY="http://proxy.studio.internal:8080" \
  -e NO_PROXY="localhost,127.0.0.1,.internal" \
  ghcr.io/pillar-interactive/pillar-agent@sha256:<verified-release-digest>

2.Verifying SAS HBA & External JBOD Controller Access

Ensure external SAS expander bays are enumerated and accessible via SCSI generic drivers:

bash

# Identify the storage controller; confirm firmware mode with vendor tooling
lspci -nn | grep -i -E "lsi|broadcom|sas"

Verify device discovery inside the agent container:

bash

# Test raw device scan inside container
docker exec -it pillar-agent smartctl --scan

3.Preflight POSIX Exit Code Taxonomy

When running ./pillar-agent --preflight, the binary exits with deterministic POSIX codes for automated script orchestration:

Exit Code	Identifier	Meaning	Remediation Action
0	ExitSuccess	Preflight suite passed nominally	Nominal. System ready to deploy.
10	ExitVfsFailure	/proc/diskstats unreadable or missing	Ensure mount: -v /proc/diskstats:/proc/diskstats:ro
11	ExitSysfsMasked	/sys/block hierarchy masked	Ensure mount: -v /sys/block:/sys/block:ro
30	ExitHardwareNoDrives	Zero physical media discovered on bus	Check HBA SAS cabling and JBOD power
40	ExitNetworkDnsFailure	Unable to resolve telemetry FQDN	Check /etc/resolv.conf and DNS routes
41	ExitNetworkTlsHandshake	TLS 1.3 handshake or CA validation failed	Check for intercepting proxy CA; mount corporate cert if required
50	ExitProtocolVersionUnsupported	Reserved. Current Agents no longer exit on HTTP 426; they log HTTP 426 Upgrade Required and retry with backoff capped at 10 minutes. The Cloud returns 426 once a release newer than the Agent has been out for more than 72 hours	Run sudo pillar update all.
60	ExitWatchdogOOMCeiling	Memory RSS > 50MB ceiling exceeded	Self-terminated to preserve host memory. Report bug.
61	ExitWatchdogCpuSpike	CPU utilization > 2.0% ceiling exceeded	Self-terminated to preserve storage I/O. Report bug.

Update and verify components

Use the signed Pillar CLI installed by the pairing flow. Scheduling depends on the host; check installer output when systemd is unavailable. Review logs, reported versions, and the first fresh heartbeat after an update. Confirm Observer health separately when it is installed.

bash

sudo pillar update all --check --json
sudo pillar update all
sudo pillar status --json
sudo pillar logs agent --tail 100

4.Graceful Uninstallation & Node De-Provisioning

Managed uninstall handles update units and containers together. Dashboard notification is best-effort: inspect data.agent.dashboard in the JSON result and revoke the node in Settings → Agent pairing if notification failed. Normal notification marks the node uninstalled and retains its key; --purge also requests revocation. Retained pairing can be reused only if the key has not otherwise been revoked. Use --purge only for permanent credential/state removal. The shared IPC volume and CLI binary can remain. Uninstalling does not erase retained cloud data or cancel billing. Legacy or custom installations require a separate inventory of their supervision and persistent state.

bash

# Remove managed components; retain local credential volume and lifecycle state
sudo pillar uninstall all --yes --json

# For permanent removal, choose --purge to also delete credentials and state:
# sudo pillar uninstall all --yes --purge --json

← Runbooks