Diagnostics & Operations

Troubleshooting & Agent Lifecycle

Detailed procedures for diagnosing corporate firewall SSL proxy interception, verifying SAS expander raw I/O, consulting POSIX exit codes, and performing graceful de-provisioning.

1. Diagnosing Firewall SSL / Deep Packet Inspection (DPI)

Enterprise security perimeters (Palo Alto, Fortinet, Zscaler) often decrypt outbound port 443 HTTPS traffic using a private internal CA certificate. This breaks TLS validation inside isolated containers.

Run this probe from the storage host to test the certificate authority issuer:

bash

curl -vI --tls-max 1.3 https://telemetry.pillarinteractive.com/api/v1/nodes/health-check

If your firewall requires an explicit forward proxy:

bash

docker run -d \
  --name pillar-agent \
  ... \
  -e HTTPS_PROXY="http://proxy.studio.internal:8080" \
  -e NO_PROXY="localhost,127.0.0.1,.internal" \
  ghcr.io/pillarinteractive/pillar-agent:latest

2. Verifying SAS HBA & External JBOD Controller Access

Ensure external SAS expander bays are enumerated and accessible via SCSI generic drivers:

bash

# Verify LSI / Broadcom SAS HBA controller in IT mode
lspci -nn | grep -i -E "lsi|broadcom|sas"

Verify device discovery inside the agent container:

bash

# Test raw device scan inside container
docker exec -it pillar-agent smartctl --scan

3. Preflight POSIX Exit Code Taxonomy

When running ./pillar-agent --preflight, the binary exits with deterministic POSIX codes for automated script orchestration:

Exit Code	Identifier	Meaning	Remediation Action
0	ExitSuccess	Preflight suite passed nominally	Nominal. System ready to deploy.
10	ExitVfsFailure	/proc/diskstats unreadable or missing	Ensure mount: -v /proc/diskstats:/proc/diskstats:ro
11	ExitSysfsMasked	/sys/block hierarchy masked	Ensure mount: -v /sys/block:/sys/block:ro
12	ExitDeviceNodeDenied	/dev nodes inaccessible	Ensure mount: -v /dev:/dev:ro
20	ExitCapSysRawioDenied	CAP_SYS_RAWIO capability missing	Add container flag: --cap-add=SYS_RAWIO
21	ExitSeccompDenied	Storage ioctl rejected by Seccomp filter	Use standard Docker seccomp or whitelist SG_IO
30	ExitHardwareNoDrives	Zero physical media discovered on bus	Check HBA SAS cabling and JBOD power
40	ExitNetworkDnsFailure	Unable to resolve telemetry FQDN	Check /etc/resolv.conf and DNS routes
41	ExitNetworkTlsHandshake	TLS 1.3 handshake or CA validation failed	Verify firewall DPI SSL bypass or add corporate cert
42	ExitNetworkPmtuBlackhole	Path MTU blackhole packet drop	Verify standard 1500 MTU on egress gateway
43	ExitNetworkProxyRejected	Enterprise proxy rejected CONNECT	Check HTTPS_PROXY credentials and ACL
50	ExitProtocolVersionUnsupported	Server returned HTTP 426 Upgrade Required	Update agent: docker pull ghcr.io/...:latest
60	ExitWatchdogOOMCeiling	Memory RSS > 50MB ceiling exceeded	Self-terminated to preserve host memory. Report bug.
61	ExitWatchdogCpuSpike	CPU utilization > 2.0% ceiling exceeded	Self-terminated to preserve storage I/O. Report bug.

4. Graceful Uninstallation & Node De-Provisioning

Pillar Agent leaves zero drivers, kernel modules, or filesystem modifications behind. Run this snippet to permanently de-provision:

bash

# 1. Stop and remove container
docker stop pillar-agent
docker rm pillar-agent

# 2. Remove persistent API key volume
docker volume rm pillar-agent-data

# 3. If deployed via systemd:
sudo systemctl disable --now pillar-agent.service
sudo rm -f /etc/systemd/system/pillar-agent.service
sudo systemctl daemon-reload