CRW-1003 - Probing Availability at the Socket Level

Living Document Notice
Published 2026-09-12. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Probing Availability at the Socket Level

Summary

Application availability checks frequently rely on external shell executions of curl or Python request scripts. Invoking language runtimes for single-shot HTTP queries introduces socket creation latency, DNS resolution variability, and unnecessary memory churn.

Crow’s Nest executes availability probes using non-blocking POSIX socket primitives. This dispatch examines deterministic HTTP GET status inspections, raw TCP SYN handshakes, socket timeout tuning, and ICMP echo probes implemented without third-party networking libraries.

The Latency Penalty of Shell-Based Probing

A common anti-pattern in host surveillance is executing curl -s -o /dev/null -w '%{http_code}' inside cron jobs every ten seconds. Spawning curl parses command-line arguments, allocates TLS configuration structs, and performs dynamic linker resolution before issuing a single SYN packet.

Probe Execution Primitive	Execution Latency	Context Switches	Heap Allocation	Zero-Copy Support
curl via /bin/sh -c subshell	14.8ms	182	2.4MB	No
Python urllib.request script	32.1ms	410	12.8MB	No
Crow’s Nest Non-Blocking Socket	0.35ms	2	0 bytes (Stack)	Yes

A monitoring node probing twenty internal microservices across the Harbormaster KPP protocol control plane and Harbor edge runtime edge caches wastes hundreds of milliseconds per cycle merely managing subprocess lifecycles.

Crow’s Nest eliminates intermediate layers by issuing direct system calls against target IP endpoints pre-resolved from the Quartermaster manifest.

Non-Blocking TCP Handshake Verification

Determining port availability requires verifying that the target TCP daemon accepts connections without completing application-layer handshakes. Crow’s Nest configures client sockets with O_NONBLOCK and interrogates connection progress via poll:

int check_tcp_port(const char *ip_str, int port, int timeout_ms) {
    int sock = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    if (sock < 0) return -1;
 
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_port = htons(port);
    inet_pton(AF_INET, ip_str, &addr.sin_addr);
 
    int res = connect(sock, (struct sockaddr *)&addr, sizeof(addr));
    if (res < 0 && errno == EINPROGRESS) {
        struct pollfd pfd = { .fd = sock, .events = POLLOUT };
        res = poll(&pfd, 1, timeout_ms);
        if (res > 0) {
            int err = 0;
            socklen_t len = sizeof(err);
            getsockopt(sock, SOL_SOCKET, SO_ERROR, &err, &len);
            close(sock);
            return (err == 0) ? 0 : -1;
        }
    }
    close(sock);
    return -1;
}

This sequence completes in sub-millisecond durations when checking localhost listeners or local rack switches. If the remote port returns an explicit TCP RST packet, poll flags POLLERR immediately, differentiating closed listeners from dropped packets.

Deterministic HTTP Status Code Parsing

HTTP probes do not need full MIME parsing or response body extraction. Crow’s Nest sends a minimal HTTP/1.1 request formatted with Connection: close and reads only the HTTP status line:

GET /healthz HTTP/1.1
 
Host: 127.0.0.1:8080
 
User-Agent: CrowsNest/1.0
 
Connection: close
 
 
 
 

The response parsing loop inspects the first twelve bytes of the socket read buffer:

// Extract HTTP response code from buffer: "HTTP/1.1 200 OK"
char rx_buf[128];
ssize_t n = recv(sock, rx_buf, sizeof(rx_buf) - 1, 0);
if (n > 12 && memcmp(rx_buf, "HTTP/1.", 7) == 0) {
    int status_code = (rx_buf[9] - '0') * 100 + 
                      (rx_buf[10] - '0') * 10 + 
                      (rx_buf[11] - '0');
    // Validate status_code == expected_status
}

Disconnecting immediately after the status line prevents buffering multi-megabyte error pages into memory. Sockets are configured with SO_RCVTIMEO set to 250 milliseconds, terminating slow responses before worker queues backup.

ICMP Echo via Linux DGRAM Sockets

Traditional ping utilities require root privileges or setuid binaries to allocate raw SOCK_RAW sockets. Crow’s Nest utilizes Linux unprivileged ICMP sockets (IPPROTO_ICMP with SOCK_DGRAM), enabled via /proc/sys/net/ipv4/ping_group_range:

int ping_sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP);
struct icmphdr icmp_req;
memset(&icmp_req, 0, sizeof(icmp_req));
icmp_req.type = ICMP_ECHO;
icmp_req.un.echo.id = htons(getpid() & 0xFFFF);
icmp_req.un.echo.sequence = htons(seq_num++);

This design allows the surveillance daemon to run under an unprivileged service account without capabilities grants (CAP_NET_RAW).

Round-trip latency is captured using kernel socket timestamps:

struct timeval tv;
ioctl(ping_sock, SIOCGSTAMP, &tv);

Operators verify probe execution times against target nodes:

crows-nest probe --target 127.0.0.1:8080 --type http --timeout 200

---

• Directus Target: crows-nest
• Garden Source Reference: MOC - Ingestion & Capture
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [CRW-1003 - Probing Availability at the Socket Level](CRW-1003 - Probing Availability at the Socket Level)