CRW-1006 - Dispatching Breaches to On The Line

Living Document Notice
Published 2026-09-15. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Dispatching Breaches to On The Line

Summary

Monitoring systems must reliably transmit confirmed operational anomalies to incident management platforms without dropping events during network isolation. Direct outbound email notifications and unstructured webhooks fail silently when local host routing tables degrade.

Crow’s Nest dispatches verified metric breaches to On The Line using structured JSON payloads over durable local domain sockets and HTTP webhooks. This dispatch details the payload contract, HMAC-SHA256 signature verification, local spooling queues, retry backoff algorithms, and idempotency guarantees.

The Dispatch Contract

When an operational check crosses an alert threshold and satisfies sliding-window persistence requirements, Crow’s Nest formats a dispatch payload. The payload provides deterministic forensic context: exact metric values, window capacity, node identification from Quartermaster, and timestamp boundaries.

{
  "event_id": "evt_01J8N6Z8P8Q9R3V4W5X6Y7Z8",
  "source": "crows-nest",
  "node_id": "vps-lon-02",
  "check_name": "harbormaster_http_5xx",
  "status": "BREACH",
  "severity": "CRITICAL",
  "timestamp": "2026-09-15T10:14:32Z",
  "metrics": {
    "target": "http://127.0.0.1:8080/healthz",
    "expected_status": 200,
    "observed_status": 503,
    "window_n": 6,
    "failed_samples": 5,
    "consecutive_failures": 4
  },
  "context": {
    "load_avg_1m": 4.12,
    "mem_available_mb": 84,
    "disk_used_pct": 72.4
  }
}

The embedded context map incorporates node vitals captured at the exact second of failure, eliminating manual log correlation during initial triage.

Breach Severity Mapping and Route Classification

Incident severity dictates downstream alerting behavior on the On The Line dispatch ledger:

Severity Level	Trigger Threshold Definition	Destination Sink	SLA Dispatch Horizon
CRITICAL	Complete service failure, HTTP 5xx persistent breach	Pager duty ledger + local bell	Immediate (< 1 sec)
WARN	Resource threshold approached (e.g. disk > 90%)	Daily triage ledger	Within 15 minutes
ADVISORY	Tender sidecar offline > lease deadline	Activity log only	Low priority queue

Severity mappings are evaluated deterministically at the moment of breach formulation.

Durable Local Spooling via FIFO Queues

Target incident endpoints can become unreachable during broad network partitioning or DNS failure. Crow’s Nest buffers alerts locally using an append-only JSON Lines file located at /var/spool/crows-nest/dispatch.queue.

+-----------------------------------------------------------+
|               Crow's Nest Dispatch Topology               |
|                                                           |
|  [Sliding Window Engine]                                  |
|            |                                              |
|            v                                              |
|  [Local Spool Queue] -----> [Dispatcher Worker Thread]    |
|  (/var/spool/...queue)                 |                  |
|                                        v                  |
|                           [On The Line HTTP API]          |
|                           (or Local Domain Socket)        |
+-----------------------------------------------------------+

A dedicated worker thread drains the spool queue:

1. Opens /var/spool/crows-nest/dispatch.queue in read/write mode.
2. Posts the leading entry to On The Line via HTTP POST /v1/incidents.
3. Upon receiving HTTP 200 OK or 201 Created, truncates the entry from the spool.
4. On network failure (ECONNREFUSED, ETIMEDOUT), backs off exponentially ($2^k\times 1000\text{ms}$) up to 60 seconds while retaining the on-disk record.
5. Issues fdatasync() on write batches to ensure queue survival during kernel panic or abrupt node reboot.

Socket-Based Local Delivery and Idempotency Keys

When On The Line operates on the same host node, network hops are bypassed entirely. Crow’s Nest delivers alert frames over a UNIX domain datagram socket (/run/on-the-line/dispatch.sock):

int send_local_dispatch(const char *json_payload, size_t len) {
    int sock = socket(AF_UNIX, SOCK_DGRAM, 0);
    if (sock < 0) return -1;
 
    struct sockaddr_un addr;
    memset(&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    strncpy(addr.sun_path, "/run/on-the-line/dispatch.sock", sizeof(addr.sun_path) - 1);
 
    ssize_t sent = sendto(sock, json_payload, len, 0, (struct sockaddr *)&addr, sizeof(addr));
    close(sock);
    return (sent == (ssize_t)len) ? 0 : -1;
}

Unix datagram delivery requires zero TCP connection state, incurs minimal latency, and functions when external routing interfaces are down.

To prevent ticket duplication during repeated evaluations, Crow’s Nest generates a deterministic SHA-256 de-duplication hash:

$\text{IdempotencyKey}=\text{SHA256}(\text{node\_id}\parallel \text{check\_name}\parallel \text{status})$

On The Line matches this hash to update existing ledger tickets rather than creating distinct events. Operators inspect current queue status directly:

# Verify dispatch queue backlog
wc -l /var/spool/crows-nest/dispatch.queue

---

• Directus Target: crows-nest
• Garden Source Reference: MOC - Ingestion & Capture
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [CRW-1006 - Dispatching Breaches to On The Line](CRW-1006 - Dispatching Breaches to On The Line)