CRW-1007 - Auto-Configuring Probes from Quartermaster

Living Document Notice
Published 2026-09-16. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Auto-Configuring Probes from Quartermaster

Summary

Static monitoring configuration files require manual updates every time a fleet node is provisioned, re-addressed, or decommissioned. Stale target inventories cause monitoring daemons to probe phantom IP addresses while leaving new nodes unmonitored.

Crow’s Nest ingests Quartermaster hardware and service manifests to dynamically construct probe lists, target thresholds, and network paths. This dispatch details declarative manifest parsing, atomic configuration reloading via SIGHUP, DNS cache pinning, and socket draining safeguards.

The Inventory Drift Problem

In multi-node infrastructure, hardware specifications and network roles evolve asynchronously. Maintaining separate inventory files for deployment tooling and telemetry daemons inevitably produces drift:

• Node IP addresses change during VPS migration, but monitoring targets continue querying decommissioned routes.
• Disk volume sizes increase from 40GB to 120GB, yet monitoring alerts remain pinned to obsolete 35GB warning limits.
• Service endpoints shift ports, generating continuous false-positive connection refuels.

Quartermaster acts as the authoritative hardware manifest for the Bosun ecosystem. Crow’s Nest reads this manifest directly, deriving surveillance rules from actual machine configurations.

Quartermaster Manifest Schema

Quartermaster exports a structured YAML inventory file defining host roles, network interfaces, and operational bounds:

# /etc/quartermaster/manifest.yaml
version: 1
nodes:
  - id: edge-fra-01
    ipv4_private: 10.10.40.12
    ipv4_public: 198.51.100.44
    specs:
      ram_mb: 2048
      disk_gb: 50
    services:
      - name: harbor-edge
        type: http
        port: 8080
        health_path: /healthz
        expected_status: 200
        timeout_ms: 300
      - name: ssh-admin
        type: tcp
        port: 22
        timeout_ms: 150
    alert_rules:
      disk_used_max_pct: 88.0
      ram_available_min_mb: 256

Crow’s Nest maps each service entry into an in-memory probe descriptor:

Manifest Field	Generated Probe Type	Interval	Window Constraint	Socket Options
health_path: /healthz	HTTP Status Probe	10s	4 of 6 failures	SO_RCVTIMEO (300ms)
port: 22	TCP SYN Connect	30s	2 of 3 failures	O_NONBLOCK
disk_used_max_pct	/proc/diskstats + statvfs	60s	5 of 5 failures	Direct VFS poll

Target IP addresses are pinned directly in memory, bypassing recurring glibc getaddrinfo lookups that stall collection loops during upstream DNS latency spikes.

Atomic Configuration Reload via POSIX Signals

Surveillance daemons must not terminate active monitoring loops or drop inflight state counters when updating configuration rules. Crow’s Nest reloads target lists atomically upon receiving SIGHUP:

static volatile sig_atomic_t g_reload_requested = 0;
 
void handle_sighup(int sig) {
    (void)sig;
    g_reload_requested = 1;
}
 
// Inside main event loop:
if (g_reload_requested) {
    g_reload_requested = 0;
    ConfigState *new_cfg = load_quartermaster_manifest("/etc/quartermaster/manifest.yaml");
    if (new_cfg != NULL) {
        ConfigState *old_cfg = g_active_config;
        g_active_config = new_cfg; // Atomic pointer swap
        free_config_state(old_cfg);
        syslog(LOG_INFO, "Quartermaster manifest reloaded successfully");
    }
}

The atomic pointer swap ensures that probe worker threads always read consistent configuration structs without locking contention. Existing socket descriptors associated with retired target endpoints are drained and closed cleanly.

Validating Target Sanity and IP Strings

To prevent corrupt manifest edits from bricking the monitoring daemon, the parser validates candidate configurations against safety rules:

1. timeout_ms must not exceed poll_interval_seconds * 1000.
2. Target IP addresses must be strictly validated before insertion into probe tables:

int validate_ip_address(const char *ip_str) {
    struct in_addr sa;
    struct in6_addr sa6;
    if (inet_pton(AF_INET, ip_str, &sa) == 1) return 4;
    if (inet_pton(AF_INET6, ip_str, &sa6) == 1) return 6;
    return -1; // Malformed address string
}

3. At least one probe target must be present.
4. Warning thresholds must not contradict physical hardware bounds reported in Quartermaster specs.

If validation fails, Crow’s Nest rejects the reload, issues an alert to syslog, and retains the existing valid configuration. Operators trigger and verify manifest ingestion from the command line:

crows-nest config --test --manifest /etc/quartermaster/manifest.yaml
kill -HUP $(pgrep crows-nest)

---

• Directus Target: crows-nest
• Garden Source Reference: MOC - Ingestion & Capture
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [CRW-1007 - Auto-Configuring Probes from Quartermaster](CRW-1007 - Auto-Configuring Probes from Quartermaster)