BSN-1024 - Content-Hash Deduplication Without Silent Data Loss

Living Document Notice
Published 2026-09-13. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Content-Hash Deduplication Without Silent Data Loss

Summary

Operating system file notification subsystems frequently flood event queues during routine file operations. A single editor save action generates multiple asynchronous notifications spanning attribute modifications, directory touches, and byte flushes. Naively dropping duplicate events discards operational audit history, while naively re-indexing every event causes severe disk thrashing. Bosun PKM implements content-hash deduplication that preserves complete audit evidence without triggering redundant AST re-parsing.

The File Watcher Storm Problem

When a user modifies a note in a desktop editor, kernel-level file notification APIs (inotify on Linux, FSEvents on macOS, ReadDirectoryChangesW on Windows) emit clusters of events in rapid succession:

Kernel Event Sequence on Save:
1. IN_MODIFY        -> File bytes flushed to disk
2. IN_ATTRIB        -> Timestamp / chmod permissions updated
3. IN_CLOSE_WRITE   -> File handle closed
4. IN_MODIFY        -> Metadata touched by background backup utility

Processing each notification as an independent mutation forces four separate AST parsing cycles and four database write transactions for a single edit. Conversely, discarding duplicate events in memory without persistent evidence creates dark zones in diagnostic logs when troubleshooting file synchronization conflicts.

The Deduplication Ledger

The storage subsystem intercepts every filesystem notification, computes the SHA-256 digest of the target file, and consults the dedup_ledger table:

def process_file_event(self, event_path: Path) -> OutboxOperation:
    content_hash = self.compute_sha256(event_path)
    ledger_entry = self.projection.get_dedup_entry(content_hash)
 
    if ledger_entry is not None:
        # Event is a content duplicate of an already indexed state
        op = self.outbox.create_operation(
            path=str(event_path.relative_to(self.locker_root)),
            event_type="modify",
            status="committed",
            dedup_status="duplicate_suppressed",
            content_hash=content_hash,
            metadata={
                "first_seen_path": ledger_entry.first_seen_path,
                "first_seen_op_id": ledger_entry.first_seen_op_id,
                "duplicate_count": ledger_entry.duplicate_count + 1,
            },
        )
        self.projection.increment_dedup_count(content_hash)
        return op
 
    # Genuine novel content mutation
    op = self.outbox.create_operation(
        path=str(event_path.relative_to(self.locker_root)),
        event_type="modify",
        status="processing",
        dedup_status="novel_content",
        content_hash=content_hash,
    )
    self.projection.record_novel_content(content_hash, str(event_path), op.op_id)
    self.reindex_note(event_path, content_hash)
    op.transition_to_committed()
    return op

When an event matches an existing content digest, it receives dedup_status = "duplicate_suppressed". The operation is permanently committed to outbox.jsonl with full diagnostic metadata, but downstream AST parsing and full-text re-indexing are skipped entirely.

Memory Footprint of the In-Memory Filter

To prevent querying SQLite on every raw filesystem event, the engine maintains an in-memory 64-bit fingerprint cache using a bounded LRU ring buffer:

use std::num::NonZeroUsize;
use lru::LruCache;
 
pub struct ContentHashFilter {
    // 64-bit truncated hash cache (8 bytes per entry)
    recent_hashes: LruCache<u64, u32>,
}
 
impl ContentHashFilter {
    pub fn new(capacity: usize) -> Self {
        Self {
            recent_hashes: LruCache::new(NonZeroUsize::new(capacity).unwrap()),
        }
    }
 
    pub fn is_recent_duplicate(&mut self, hash_fragment: u64) -> bool {
        if let Some(count) = self.recent_hashes.get_mut(&hash_fragment) {
            *count += 1;
            true
        } else {
            self.recent_hashes.put(hash_fragment, 1);
            false
        }
    }
}

A capacity of 10,000 entries consumes exactly 80 KB of RAM, providing instantaneous event filtering while remaining well within bounded memory budgets.

Benchmark: Burst Ingestion Load

Simulating 5,000 file save bursts (1,000 actual edits, each emitting 5 watcher notifications within 100 milliseconds):

Deduplication Strategy	Total Parsed ASTs	SQLite Write Transactions	Total Wall-Clock Time	Peak Memory (RSS)
No Deduplication	5,000 files	5,000 transactions	8,420 ms	46.2 MB
In-Memory Dropping	1,000 files	1,000 transactions	1,840 ms (Zero audit log)	12.1 MB
Ledger Deduplication	1,000 files	1,000 updates + 4,000 outbox writes	2,150 ms (Full audit trail)	14.8 MB

The deduplication ledger cuts parser CPU workload by 80% while retaining continuous forensic audit records for every operating system notification.

---

• Directus Target: bosun-pkm
• Garden Source Reference: MOC - Bosun PKM Tools, MOC - Bosun PKM Engine, MOC - The Plain-Text Longevity Standard, [BSN-1004 - Frontmatter as a Strongly Typed Schema](BSN-1004 - Frontmatter as a Strongly Typed Schema)