Stax Digital Garden

HBM-1005 - Event Streaming and Drop-on-Lag Ring Buffers

2 min read

Living Document Notice Published 2026-09-14. The evolving architecture and revisions for this dispatch live in the Stax Digital Garden.

Event Streaming and Drop-on-Lag Ring Buffers

Event Streaming and Drop-on-Lag Ring Buffers: Monochromatic ice blue phosphor P7 vector CRT macro depicting a circular drop-on-lag ring buffer with sweep pointers shedding lagging overflow packets

Pushing real-time graph updates to desktop modules via naive WebSocket or Server-Sent Events (SSE) pipelines risks unbounded memory expansion. If a connected module stalls during message consumption, unconsumed event buffers accumulate in server process memory, in practice triggering an out-of-memory crash of the host daemon.

Harbormaster resolves slow-client memory exhaustion by enforcing drop-on-lag bounded ring buffers. The event dispatcher caps per-client queues at fixed capacities. When a lagging client exceeds buffer thresholds, Harbormaster drops older messages and signals an overflow state, protecting the core daemon memory footprint.

Event Dispatch Topology and Queue Isolation

Each connected module consumes from an isolated memory buffer:

[ Harbor Knowledge Engine ] ──► Emits graph change event
                                          │
                                          ▼
                         ┌─────────────────────────────────┐
                         │  Harbormaster Event Dispatcher  │
                         └───────┬─────────────────┬───────┘
                                 │                 │
                 Bounded Queue A │                 │ Bounded Queue B (Slow)
                 (Capacity: 128) │                 │ (Overflow Triggered)
                                 ▼                 ▼
                        [ Status Widget ]      [ Lagging Client ]
                         Receives event         Drops oldest events
                                                Signals ERR_STREAM_LAG
ParameterUnbounded Stream PatternHarbormaster Drop-on-Lag Buffer
Buffer BoundaryInfinite FIFO listFixed ring buffer (128 slots)
Slow Client ImpactConsumes host RAM until OOMDrops oldest event; logs warning
Disconnection ThresholdNever disconnects automaticallyDisconnects on persistent 30s stall
Memory FloorUnpredictable (up to hundreds of MB)Fixed ceiling (< 2 MB total across clients)

Bounded Ring Buffer Implementation

import collections
from typing import Any, Dict
 
class BoundedEventBuffer:
    def __init__(self, capacity: int = 128):
        self.capacity = capacity
        self.queue = collections.deque(maxlen=capacity)
        self.dropped_events_count = 0
 
    def push_event(self, event: Dict[str, Any]) -> bool:
        if len(self.queue) == self.capacity:
            self.dropped_events_count += 1
            # Evicts oldest automatically via deque maxlen
            self.queue.append(event)
            return False
        self.queue.append(event)
        return True
 
    def pop_event(self) -> Dict[str, Any]:
        if not self.queue:
            return None
        return self.queue.popleft()

Streaming Event Probing via cURL

Subscribe to real-time events over loopback HTTP:

# Connect to event stream with valid capability token
curl -N -H "Authorization: Bearer tok_stream_valid_089f" \
     -H "Host: 127.0.0.1:8765" \
     http://127.0.0.1:8765/protocol/v1/events/stream
  • Directus Target: harbormaster
  • Garden Source Reference: MOC - Harbormaster Protocol, MOC - Bosun PKM Tools
  • Garden Source Reference: [HBM-1005 - Event Streaming and Drop-on-Lag Ring Buffers](HBM-1005 - Event Streaming and Drop-on-Lag Ring Buffers)

Graph View