CRW-1009 - Auditing Outrigger Sandboxed Execution

Living Document Notice
Published 2026-09-18. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Auditing Outrigger Sandboxed Execution

Summary

Autonomous coding agents and untrusted data extraction pipelines can exhaust host memory, leak file descriptors, or enter runaway infinite loops. Monitoring host-level averages fails to identify runaway processes contained inside localized security perimeters.

Crow’s Nest interfaces with the Outrigger Protocol sandboxing runtime to audit per-container resource ceilings, CPU time limits, seccomp filters, and non-zero termination statuses. This dispatch details cgroups v2 accounting, kill-signal auditing, and execution ledger tracking.

The Blind Spot of Host-Level Telemetry

Host-level CPU and memory averages conceal localized sandboxed failures. An agent running inside an isolated sandbox might crash due to an out-of-memory error (SIGKILL via cgroup limiter), but if the host maintains 8GB of free memory, host-level metrics show no anomaly.

To provide real operational visibility, telemetry must monitor the sandboxing boundary itself. Outrigger executes untrusted processes inside Linux cgroups v2 and user namespaces. Crow’s Nest polls these specific cgroup accounting interfaces directly.

Interrogating cgroups v2 Accounting Interfaces

Linux cgroups v2 exposes execution parameters under /sys/fs/cgroup/outrigger/<job_id>/:

cgroup v2 Interface	Metric Measured	Operational Concern
memory.current	Current RSS + Page Cache	Memory ceiling approach
memory.events	oom_kill and high counts	Hard execution failure
cpu.stat	usage_usec and throttled_usec	CPU quota exhaustion
pids.current	Active process count	Fork bomb detection

Crow’s Nest inspects memory.events directly:

int check_cgroup_oom(const char *job_id) {
    char path[256];
    snprintf(path, sizeof(path), "/sys/fs/cgroup/outrigger/%s/memory.events", job_id);
    int fd = open(path, O_RDONLY);
    if (fd < 0) return -1;
 
    char buf[512];
    ssize_t n = read(fd, buf, sizeof(buf) - 1);
    close(fd);
    if (n <= 0) return -1;
    buf[n] = '\0';
 
    char *ptr = strstr(buf, "oom_kill ");
    if (ptr) {
        unsigned long oom_kills = strtoul(ptr + 9, NULL, 10);
        return (oom_kills > 0) ? 1 : 0;
    }
    return 0;
}

CPU Quota Throttling Metrics

When an autonomous task exhausts its allocated CPU bandwidth, the kernel scheduler delays execution by placing threads in wait states. Crow’s Nest measures throttling severity using delta calculations from cpu.stat:

$\text{Throttling}\%=100.0\times \frac{\Delta \text{throttled\_usec}}{\Delta \text{usage\_usec}+\Delta \text{throttled\_usec}}$

If throttling exceeds 30% across three consecutive intervals, Crow’s Nest logs a scheduling contention warning to the incident ledger, alerting operators that worker tasks require higher quota ceilings.

Process Exit Code and Signal Audit

When an Outrigger sandbox task finishes, Crow’s Nest reads the termination summary recorded by the process supervisor:

{
  "job_id": "outrigger_task_8819b",
  "command": "python3 /workspace/extract_data.py",
  "exit_code": 137,
  "signal": "SIGKILL",
  "runtime_ms": 4120,
  "peak_memory_bytes": 536870912,
  "cgroup_oom_triggered": true
}

Exit code 137 indicates abnormal termination via signal 9. Correlating this with cgroup_oom_triggered: true allows Crow’s Nest to report the exact cause of death without manual core dump analysis.

Exit Code	Signal	Root Cause
137	SIGKILL	cgroup memory ceiling exceeded
139	SIGSEGV	Illegal memory address violation
152	SIGXCPU	Exceeded wall-clock execution limit
159	SIGSYS	Blocked system call (Seccomp BPF filter)

Enforcing Fleet Safety Ceilings

When multiple agent tasks execute concurrently, Crow’s Nest aggregates sandbox consumption against fleet limits:

$\text{TotalSandboxRAM}={\sum }_{j\in \text{Jobs}}{\text{RAM}}_j<0.65\times {\text{HostRAM}}_{\text{total}}$

If sandbox consumption exceeds 65% of physical memory, Crow’s Nest signals the Outrigger orchestrator to pause queued job dispatches, preventing sandbox processes from consuming memory allocated to the host kernel or network proxies:

crows-nest outrigger --audit --active-jobs

The daemon records every completed run into an append-only JSON Lines ledger, capturing peak memory RSS, CPU cycles consumed, and wall-clock execution latency for forensic analysis.

---

• Directus Target: crows-nest
• Garden Source Reference: MOC - Ingestion & Capture
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [CRW-1009 - Auditing Outrigger Sandboxed Execution](CRW-1009 - Auditing Outrigger Sandboxed Execution)