CRW-1008 - Collecting Tender Sidecar Heartbeats

Living Document Notice
Published 2026-09-17. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Collecting Tender Sidecar Heartbeats

Summary

Remote workstations and mobile nodes do not run static public IP addresses or maintain constant network connectivity. Probing these nodes with external TCP checks generates continuous false alarms whenever a device sleeps or switches network interfaces.

Crow’s Nest tracks intermittent nodes by passively collecting outbound heartbeat signals from the Tender sidecar daemon. This dispatch details heartbeat payloads, cryptographic identity verification via Ed25519, monotonic lease expirations, and offline sync backlog tracking.

Passive Ingestion vs Active Probing

Active probing functions well for data center servers with permanent static routing. It fails completely when applied to client workstations, edge laptops, or intermittent worker machines:

• Nodes sit behind carrier-grade NAT (CGNAT) or home routers without port forwarding.
• Laptops enter low-power sleep modes, dropping network interfaces legitimately.
• Wi-Fi handover causes transient route breaks lasting several seconds.

Crow’s Nest handles intermittent infrastructure by inverting the surveillance model: nodes push lightweight heartbeat frames into Crow’s Nest collectors via the local Tender sidecar daemon.

Tender Heartbeat Payload Structure

Tender emits an ultra-compact binary or JSON heartbeat frame on fixed intervals (e.g., every 60 seconds):

{
  "node_id": "macbook-air-m2-remote",
  "client_version": "tender/0.8.2",
  "ts": 1773580800,
  "lease_seconds": 180,
  "vitals": {
    "battery_pct": 84,
    "charging": true,
    "disk_free_gb": 142.6,
    "offline_journal_depth": 14
  }
}

The critical metric for PKM data integrity is offline_journal_depth. If a remote node accumulates thousands of un-synced note revisions without flushing to the hub, Crow’s Nest flags potential data drift.

Heartbeat Cadence and Hardware States

Tender dynamically adapts its transmission frequency based on power source and network type:

Host Power State	Heartbeat Interval	Lease Window Duration	Backlog Alert Threshold
AC Mains Connected	30 seconds	90 seconds	> 100 journal entries
Battery Powered (> 20%)	60 seconds	180 seconds	> 250 journal entries
Low Power Mode (< 20%)	300 seconds	900 seconds	> 500 journal entries

This tiered schedule reduces radio wakeup power consumption on battery-operated edge devices while maintaining visibility over pending sync transactions.

Monotonic Lease Expiration Tracking

Crow’s Nest maintains an in-memory lease table for registered Tender sidecars. Each node is assigned a deadline:

${\text{Deadline}}_i=\text{MonotonicTime}+\text{lease\_seconds}$

A background sweep thread executes every ten seconds:

typedef struct {
    char node_id[64];
    uint64_t deadline_ns;
    uint32_t journal_depth;
    uint8_t is_expired;
} TenderLeaseEntry;
 
void sweep_leases(TenderLeaseEntry *table, size_t count, uint64_t now_ns) {
    for (size_t i = 0; i < count; i++) {
        if (!table[i].is_expired && now_ns > table[i].deadline_ns) {
            table[i].is_expired = 1;
            // Emit state change to On The Line
            dispatch_tender_offline(table[i].node_id);
        }
    }
}

Because leases are tracked using CLOCK_MONOTONIC, system wall-clock adjustments (NTP steps) do not trigger phantom lease cancellations.

Cryptographic Authentication and Flapping Prevention

Inbound heartbeat frames carry a detached Ed25519 signature verified against node public keys enrolled in Quartermaster:

// Verify Ed25519 node signature
int verify_heartbeat_sig(const uint8_t *payload, size_t len, const uint8_t *sig, const uint8_t *pubkey) {
    return ed25519_verify(sig, payload, len, pubkey);
}

Signature validation runs over a SHA-256 digest of the canonical JSON byte string. If the cryptographic signature fails validation, the frame is dropped immediately and an unauthenticated probe attempt is logged with the offending client IP. This verification prevents network attackers from injecting fraudulent node status reports into the telemetry collector.

When a node checks in after an extended disconnection, Crow’s Nest updates the registry state from OFFLINE to ONLINE without dispatching panic notifications. If offline_journal_depth > 500, the daemon emits a low-priority synchronization notice to track database compaction progress.

# Inspect registered Tender sidecar heartbeats
crows-nest tender --list --active

---

• Directus Target: crows-nest
• Garden Source Reference: MOC - Ingestion & Capture
• Garden Source Reference: MOC - Fleet Operations
• Garden Source Reference: MOC - Bosun PKM Tools
• Garden Source Reference: [CRW-1008 - Collecting Tender Sidecar Heartbeats](CRW-1008 - Collecting Tender Sidecar Heartbeats)