OTL-1004 - Idempotent Disaster Recovery Drills with Btrfs Snapshot Trees

Living Document Notice
Published 2026-09-13. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Idempotent Disaster Recovery Drills with Btrfs Snapshot Trees

Summary

Recovery runbooks that are not validated under simulated corruption inevitably fail during real emergency incidents. Structuring vault backups and Fleet Operations as immutable Btrfs snapshot trees enables automated, non-destructive recovery verification by mounting read-only subvolumes in transient namespaces.

The Fallacy of Unverified Backups

A backup archive that has never been mounted and verified represents an unverified assumption, not a recovery plan. Traditional backup drills are performed infrequently because restoring live state to production volumes risks overwriting recent transactions or causing accidental data loss.

Using Btrfs copy-on-write subvolume semantics, recovery drills run directly on real production storage snapshots without risking active datasets. Read-only snapshot trees decouple backup extraction from live operational branches.

+-------------------------------------------------------------+
|                Btrfs Snapshot Drill Pipeline                |
|                                                             |
|   [Active Data: /var/lib/bosun/vault]                       |
|           |                                                 |
|           +--- btrfs subvolume snapshot -r                  |
|           |                                                 |
|           v                                                 |
|   [Read-Only Snapshot: /snapshots/vault-20260911T0000Z]     |
|           |                                                 |
|           +--- btrfs subvolume snapshot (writable clone)    |
|           |                                                 |
|           v                                                 |
|   [Isolated Namespace: /tmp/drill-sandbox]                  |
|           |                                                 |
|           v                                                 |
|   [SQLite PRAGMA integrity_check + File Manifest BLAKE3]    |
|           |                                                 |
|           v                                                 |
|   [Cleanup Trap: btrfs subvolume delete --commit]           |
+-------------------------------------------------------------+

Atomic Snapshot Ingestion Workflow

Daily backups generate an immutable, read-only snapshot of the production subvolume. Btrfs completes this operation in constant time by duplicating root b-tree pointers:

# Create immutable point-in-time reference
$ btrfs subvolume snapshot -r /var/lib/bosun/vault /snapshots/vault-2026-09-11-0000
 
# Generate BLAKE3 root tree manifest for cryptographic attestation
$ b3sum /snapshots/vault-2026-09-11-0000/manifest.json > /snapshots/vault-2026-09-11-0000/manifest.b3

Because snapshots share underlying physical extents with the live filesystem, generating a snapshot incurs zero disk I/O overhead until subsequent modifications occur on the source volume.

Automated Verification Suite

The validation suite spins up an ephemeral mount point, checks SQLite index integrity, verifies file system consistency, and tears down the scratch volume:

#!/usr/bin/env bash
set -euo pipefail
 
SNAPSHOT_DIR="/snapshots/vault-2026-09-11-0000"
DRILL_ROOT="/mnt/drills/sandbox-$(date +%s)"
 
cleanup() {
    echo "Tearing down recovery sandbox..."
    if [ -d "$DRILL_ROOT" ]; then
        btrfs subvolume delete "$DRILL_ROOT" > /dev/null 2>&1 || true
        rmdir "$DRILL_ROOT" || true
    fi
}
trap cleanup EXIT INT TERM
 
echo "Creating writable clone from read-only snapshot..."
mkdir -p "$DRILL_ROOT"
btrfs subvolume snapshot "$SNAPSHOT_DIR" "$DRILL_ROOT"
 
echo "Executing SQLite structural integrity verification..."
SQLITE_DB="$DRILL_ROOT/state/index.sqlite3"
if [ -f "$SQLITE_DB" ]; then
    INTEGRITY_RESULT=$(sqlite3 "$SQLITE_DB" "PRAGMA quick_check;")
    if [ "$INTEGRITY_RESULT" != "ok" ]; then
        echo "CRITICAL: Database integrity verification failed: $INTEGRITY_RESULT" >&2
        exit 1
    fi
    echo "SQLite integrity verification: OK"
else
    echo "CRITICAL: Database file missing from snapshot" >&2
    exit 1
fi
 
echo "Verifying document manifest checksums..."
if [ -f "$DRILL_ROOT/manifest.b3" ]; then
    (cd "$DRILL_ROOT" && b3sum --check manifest.b3 --quiet)
    echo "Manifest checksums: VERIFIED"
fi
 
echo "Disaster recovery drill passed successfully."

Failure Diagnosis Matrix

When an automated drill fails, the validation exit code points directly to the failure domain:

Exit Code	Failure Stage	Root Cause Indicator	Remediating Action
1	Database Quick Check	SQLite B-tree page corruption or mid-flush write	Inspect WAL log flush configuration; verify fsync flags.
2	Manifest Hash Mismatch	Silent bit rot or incomplete snapshot synchronization	Run btrfs scrub across physical array devices.
3	Snapshot Clone Failure	Storage pool quota exhaustion or inode table saturation	Inspect btrfs filesystem usage /mnt/storage.

Running non-destructive recovery verification on every daily snapshot guarantees that the restoration procedure works before an actual hardware failure or data corruption event occurs.

---

• Directus Target: on-the-line
• Garden Source Reference: idempotent-disaster-recovery-drills-with-btrfs-snapshot-trees, btrfs-integrity-verification, backup-validation-runbooks, storage-durability, MOC - Fleet Operations, MOC - Bosun PKM Tools