Stax Digital Garden

QTM-1010 - The Reality of Configuration Drift (Kernel updates, hotfix drift, and reconciling manifests)

4 min read

Living Document Notice
Published 2026-09-19. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

The Reality of Configuration Drift (Kernel updates, hotfix drift, and reconciling manifests)

The Reality of Configuration Drift (Kernel updates, hotfix drift, and reconciling manifests): Emerald green P31 baseline reference waveform with warm golden-amber P20 diverging configuration drift trace and reconciliation brackets

Summary

Declared system manifests inevitably diverge from running state when security patches apply, kernel modules load, or manual hotfixes execute during outages. Unmonitored configuration drift invalidates scheduler assumptions and compromises fleet predictability. Quartermaster implements an audit pipeline that continuously detects discrepancies between declared manifests and kernel runtime state.

The Inevitability of Operational Drift

In immutable infrastructure models, servers are terminated and reprovisioned whenever configurations change. On physical bare-metal nodes and economical hosting setups, recreating machines from scratch for every kernel security patch or sysctl tuning parameter is impractical. Nodes have persistent operational lifetimes measured in years.

Over time, discrepancies emerge between the declared manifest in Git and the actual runtime state of the host. An operator temporarily adjusts net.core.somaxconn during a traffic spike and forgets to record the change. A security update silently replaces a shared library or upgrades the Linux kernel during an unattended reboot.

Treating drift as an operational crime leads to unrecorded changes. Quartermaster treats drift as a normal state property requiring continuous, non-destructive measurement.

Non-Destructive Reconciliation Philosophy

Automated configuration management tools often practice aggressive auto-remediation: if a setting differs from the playbook, the tool immediately overwrites it. In production, this behavior can turn a controlled emergency hotfix into an immediate cascade outage.

Quartermaster enforces a strict separation between auditing and remediation:

  • Auditing runs continuously as a read-only process, comparing live system values against the declared YAML manifest.
  • Discrepancies generate structured drift events exported to telemetry systems.
  • Manifest updates or host reconciliations require explicit operator initiation through version control commits.

Configuration Domain Drift Matrix

Quartermaster inspects distinct operating system layers, categorizing discrepancies by operational impact.

System DomainLive Audit Inspection SourceDeclared Manifest KeyDrift ImpactRemediation Workflow
Kernel Release/proc/sys/kernel/osreleasesystem.kernel_versionHigh: unexpected ABI changes or module breakageSchedule reboot into declared kernel or update manifest
Sysctl Values/proc/sys/net/*, /proc/sys/fs/*system.sysctl_parametersMedium: degraded network buffers or link limitsApply manifest baseline via sysctl -p
Block Devices/sys/block/*/queue/schedulerstorage.io_schedulerLow: suboptimal disk throughput under loadEcho declared scheduler into sysfs queue node
Network Routing/sbin/ip route shownetworking.routesCritical: traffic blackholing or route flappingAudit routing table; commit permanent route to network config
Package State/var/lib/dpkg/status or apk info -vpackages.installed_manifestHigh: untested package versions deployedReconcile package version pins in deployment repo

Automated Drift Detection Script

The drift auditing tool executes locally on each node, generating a structured diff without altering system state.

#!/bin/sh
set -eu
 
MANIFEST="/etc/quartermaster/node-spec.yaml"
AUDIT_LOG="/var/log/quartermaster/drift.log"
DRIFT_FOUND=0
 
mkdir -p "$(dirname "$AUDIT_LOG")"
echo "=== Quartermaster Drift Audit: $(date -u +'%Y-%m-%dT%H:%M:%SZ') ===" > "$AUDIT_LOG"
 
# 1. Audit kernel release
LIVE_KERNEL=$(uname -r)
DECLARED_KERNEL=$(awk -F': ' '/kernel_version:/ {print $2}' "$MANIFEST" | tr -d ' "')
 
if [ "$LIVE_KERNEL" != "$DECLARED_KERNEL" ]; then
  echo "DRIFT: Kernel mismatch [Live: $LIVE_KERNEL | Declared: $DECLARED_KERNEL]" >> "$AUDIT_LOG"
  DRIFT_FOUND=1
fi
 
# 2. Audit critical sysctl parameters
for PARAM in "net.ipv4.tcp_syncookies" "fs.protected_hardlinks"; do
  LIVE_VAL=$(sysctl -n "$PARAM" 2>/dev/null || echo "missing")
  DECLARED_VAL=$(grep "$PARAM" "$MANIFEST" | awk -F'= ' '{print $2}' | tr -d ' "')
 
  if [ -n "$DECLARED_VAL" ] && [ "$LIVE_VAL" != "$DECLARED_VAL" ]; then
    echo "DRIFT: Sysctl $PARAM [Live: $LIVE_VAL | Declared: $DECLARED_VAL]" >> "$AUDIT_LOG"
    DRIFT_FOUND=1
  fi
done
 
# 3. Check for unauthorized listening network ports
LIVE_PORTS=$(ss -tlpn | awk 'NR>1 {print $4}' | cut -d: -f2 | sort -u | tr '
' ' ')
echo "Active Listening Ports: $LIVE_PORTS" >> "$AUDIT_LOG"
 
if [ "$DRIFT_FOUND" -eq 1 ]; then
  echo "AUDIT FAILED: Discrepancies detected between live state and $MANIFEST" >&2
  exit 1
fi
 
echo "AUDIT SUCCESS: System state conforms to manifest"

Audit Invocation and Reporting

Nodes execute drift audits periodically via cron or systemd timers, reporting results directly to monitoring logs:

# Execute local drift check
quartermaster-cli audit --manifest /etc/quartermaster/node-spec.yaml
 
# Diff live kernel sysfs against declared manifest baseline
quartermaster-cli diff --category sysctl --verbose
  • Directus Target: quartermaster
  • Garden Source Reference: MOC - Fleet Operations
  • Garden Source Reference: MOC - Bosun PKM Tools
  • Garden Source Reference: [QTM-1010 - The Reality of Configuration Drift (Kernel updates, hotfix drift, and reconciling manifests)](QTM-1010 - The Reality of Configuration Drift (Kernel updates, hotfix drift, and reconciling manifests))

Graph View