TND-1007 - Heartbeats and Queue Telemetry for Crows Nest

Living Document Notice
Published 2026-11-01. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Heartbeats and Queue Telemetry for Crows Nest

Summary

Fleet visibility across distributed background daemons requires lightweight telemetry that avoids polling latency and connection overhead. Running heavy monitoring agents or establishing periodic TLS connections from background daemons wastes workstation resources and pollutes system logs. The Tender daemon emits compact binary telemetry frames over UDP to local or network Crows Nest collectors.

These telemetry frames pack uncommitted journal offsets, dirty file counts, and memory resident metrics into fixed-length datagrams. Workstation health metrics stream continuously to fleet collectors with sub-millisecond serialization latency and zero socket allocation churn.

Lightweight Telemetry Without Connection Overhead

Monitoring long-running personal knowledge daemons is necessary when managing multi-device synchronization. Operators need visibility into whether local queues are stalled, how many files remain dirty, and whether memory consumption is expanding unexpectedly.

Standard observability approaches rely on HTTP endpoints scraped by Prometheus or JSON payloads pushed over HTTPS. For background daemons designed to run unobtrusively on laptops and low-power appliances, these approaches introduce significant overhead:

1. TCP handshakes and TLS session negotiation consume CPU cycles and memory.
2. Dynamic JSON serialization generates heap allocations on every heartbeat tick.
3. Blocked HTTP calls stall internal monitoring threads during network disconnects.

The Tender daemon replaces connection-oriented metrics with compact UDP datagram heartbeats directed to a Crows Nest telemetry collector.

+--------------------------------------------------------------------+
|                  Crows Nest Telemetry Frame (32B)                  |
|                                                                    |
| 0x00: [ Magic Header: 2B (0x42, 0x53 = 'BS') ]                    |
| 0x02: [ Protocol Version: 2B (uint16 = 1) ]                        |
| 0x04: [ Daemon Uptime: 4B (uint32 seconds) ]                       |
| 0x08: [ Uncommitted Journal Offset: 8B (uint64 bytes) ]            |
| 0x10: [ Dirty File Count: 4B (uint32) ]                            |
| 0x14: [ Resident Set Size: 4B (uint32 kilobytes) ]                 |
| 0x18: [ Last Sync Timestamp: 4B (uint32 epoch seconds) ]           |
| 0x1C: [ Checksum: 4B (CRC-32C over bytes 0x00..0x1B) ]             |
+--------------------------------------------------------------------+

Binary Telemetry Frame Layout

The Crows Nest heartbeat protocol uses a fixed 32-byte binary datagram. The layout aligns on 4-byte boundaries, enabling zero-copy struct casting and parsing across varied CPU architectures:

#[repr(C, packed)]
pub struct CrowsNestHeartbeat {
    pub magic: [u8; 2],            // b"BS"
    pub version: u16,              // Protocol version
    pub uptime_secs: u32,          // Process uptime
    pub uncommitted_bytes: u64,    // Uncommitted WAL offset
    pub dirty_files: u32,          // Modified files awaiting sync
    pub rss_kb: u32,               // Resident set size in KB
    pub last_sync_epoch: u32,      // Timestamp of last clean sync
    pub crc32: u32,                // Integrity checksum
}

Every ten seconds, the daemon samples internal metrics counters, packs the values into the struct, computes the CRC-32C checksum, and transmits the datagram in a single non-blocking syscall:

ssize_t bytes_sent = sendto(
    telemetry_sock_fd,
    &heartbeat_frame,
    sizeof(heartbeat_frame),
    MSG_DONTWAIT,
    (struct sockaddr *)&collector_addr,
    sizeof(collector_addr)
);

Because UDP is connectionless and non-blocking, a missing or unreachable Crows Nest collector never delays daemon execution. If a packet drops during network transitions, the daemon simply transmits fresh metrics during the subsequent interval.

Ingestion at Crows Nest

The Crows Nest collector listens on a dedicated UDP port (9422). Upon receiving a packet, the collector validates the two magic bytes and verifies the trailing CRC-32C checksum:

Offset Range	Field Name	Validation Rule	Diagnostic Trigger
0x00..0x01	Magic Header	Must equal 0x4253	Drop packet if mismatched
0x08..0x0F	Uncommitted Offset	Value > 50 Megabytes	Alert: Local journal backlog building
0x10..0x13	Dirty File Count	Value > 500 Files	Alert: Filesystem coalescing backlog
0x14..0x17	Resident Set Size	Value > 65536 KB	Alert: Daemon memory threshold exceeded

Collectors ingest tens of thousands of heartbeats per second with minimal CPU load. The collector transforms these fixed-width frames into time-series records without JSON tokenization or string allocations:

crows_nest_collector: ingested 32B frame from node-worker-04 [dirty_files=12, rss_kb=14320, wal_lag=0B]

This telemetry pipeline maintains operational visibility across distributed vault nodes while keeping client daemons lightweight and responsive.

---

• Directus Target: tender
• Garden Source Reference: Crows Nest Collector, Fleet Telemetry Schemas, Daemon Heartbeat Protocols, MOC - Ingestion & Capture, MOC - Local-First Systems and Synchronization, MOC - Bosun PKM Tools