OTL-1003 - Deterministic Backpressure with Token Bucket Shaper in NGINX and Cgroups

Living Document Notice
Published 2026-09-12. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Deterministic Backpressure with Token Bucket Shaper in NGINX and Cgroups

Summary

Uncontrolled burst traffic can cascade through local-first sync relays and exhaust kernel page cache buffers before operators can diagnose upstream stalls. Combining NGINX leaky bucket burst limits with Linux cgroup v2 memory high limits forces upstream sync clients to throttle gracefully before invoking the kernel OOM killer.

Traffic Bursts and Unbounded Buffer Allocation

Edge synchronization daemons receive erratic client payloads. When multiple offline clients reconnect simultaneously after an extended disruption, inbound sync streams saturate disk write bandwidth and consume server memory in unbounded buffer queues. Without strict admission control at the reverse proxy boundary, incoming connection bursts trigger kernel thrashing and cascading timeouts.

Deterministic backpressure enforces explicit rate envelopes at two distinct layers:

1. L7 Protocol Admission: NGINX limits request ingestion rate using leaky-bucket token tracking, returning explicit HTTP 429 status codes to clients with standardized retry headers.
2. L4 OS Resource Boundary: Linux cgroup v2 isolates daemon execution within strict memory and I/O slices, converting sudden memory allocation spikes into controlled page-reclamation pressure rather than sudden OOM termination.

NGINX Token Bucket Configuration

The edge gateway applies a dual-zone token bucket schema. High-frequency health probes pass unimpeded, while batch sync operations encounter rigid throughput gates:

# /etc/nginx/conf.d/backpressure.conf
http {
    # 10 megabyte zone tracking ~160,000 distinct client IPs
    limit_req_zone $binary_remote_addr zone=sync_ingest_zone:10m rate=20r/s;
    limit_req_status 429;
 
    server {
        listen 443 ssl http2;
        server_name sync.bosunpkm.com;
 
        location /api/v1/sync/ {
            # Allow short burst of 10 requests, rejecting subsequent excess without delay
            limit_req zone=sync_ingest_zone burst=10 nodelay;
            
            # Explicit client backpressure advisory header
            add_header Retry-After 5 always;
 
            proxy_pass http://127.0.0.1:8080;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
            proxy_send_timeout 10s;
            client_max_body_size 8M;
        }
 
        location /healthz {
            access_log off;
            return 200 "OK
";
        }
    }
}

When client request frequency exceeds the 20 requests/second threshold and exhausts the 10-token burst allowance, NGINX immediately terminates the handshake with an RFC 6585 code 429 Too Many Requests.

Linux Cgroups v2 Resource Isolation

To protect the host node when authenticated clients perform sustained disk writes within rate limits, systemd constrains the worker slice:

# /etc/systemd/system/bosun-sync.service.d/limits.conf
[Service]
# Enforce cgroup v2 hierarchy
Slice=system-sync.slice
 
# Memory throttles: High initiates reclaim; Max triggers strict cutoff
MemoryHigh=512M
MemoryMax=640M
MemorySwapMax=0
 
# I/O write rate ceilings to prevent NVMe write buffer exhaustion
IOWeight=100
IOReadIOPSMax=/dev/disk/by-id/nvme-eui.002538b4115038c1 2000
IOWriteIOPSMax=/dev/disk/by-id/nvme-eui.002538b4115038c1 1200
TasksMax=128

The memory throttle dynamics operate in two distinct thresholds:

Control Parameter	Threshold	Operational Consequence
MemoryHigh	512 MiB	Kernel throttles allocating threads in sys_enter; forces background page writeback.
MemoryMax	640 MiB	Invocations of direct memory reclamation; process paused if reclamation fails.
MemorySwapMax	0 MiB	Prohibits swapping; prevents silent disk thrash degradation under peak concurrency.

Runtime Inspection and Throttling Validation

Verify active backpressure state by inspecting cgroup counters during simulated burst injection:

# Verify active cgroup pressure metrics
$ systemd-cgtop -n 1 --order=memory system-sync.slice
 
# Query real-time memory pressure stalls
$ cat /sys/fs/cgroup/system.slice/system-sync.slice/memory.pressure
some avg10=0.00 avg60=0.00 avg300=0.00 total=142011
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
 
# Trigger client burst to verify HTTP 429 response enforcement
$ for i in {1..25}; do     curl -s -o /dev/null -w "%{http_code}
" https://sync.bosunpkm.com/api/v1/sync/batch &   done; wait
200
200
200
429
429
429

The combination of deterministic L7 status codes and kernel-level cgroup pressure preserves node responsiveness even during sudden synchronized reconnection events.

---

• Directus Target: on-the-line
• Garden Source Reference: deterministic-backpressure-with-token-bucket-shaper-in-nginx-and-cgroups, cgroups-v2-resource-limits, nginx-rate-limiting, flow-control, MOC - Fleet Operations, MOC - Bosun PKM Tools, MOC - Local-First Systems and Synchronization