Stax Digital Garden

BSN-1025 - Opt-In Full-Text Indexing Keeping Engine Footprint Under 20MB

3 min read

Living Document Notice
Published 2026-09-14. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Opt-In Full-Text Indexing Keeping Engine Footprint Under 20MB

Summary

Full-text indexing every document in a large personal knowledge base inflates SQLite database files, increases RAM consumption by orders of magnitude, and degrades application startup times. By enforcing an opt-in architecture where note bodies remain unindexed by default, Bosun PKM keeps its background daemon resident set size (RSS) strictly below 20 MB while preserving fast title and frontmatter searches.

The Memory Cost of Unbounded FTS

Knowledge base indexers frequently attempt to tokenize every word in a vault upon startup. For a vault containing 50,000 Markdown notes, unconstrained full-text search triggers substantial overhead:

Full-Text Index Overhead Breakdown (50,000 Notes, 200 MB Text):
1. FTS5 Index Tables:             ~180 MB - 240 MB additional SQLite disk space
2. SQLite Page Cache:              64 MB - 128 MB RAM to avoid disk thrashing
3. Tokenizer In-Memory State:      30 MB - 50 MB heap allocation
Total Engine Footprint:           > 250 MB RAM (Exceeds modest background daemon limits)

On laptops and low-power hardware, a 250 MB background daemon leads to battery drain and memory pressure. Most navigation workflows rely on title matches, tags, and wikilink traversals rather than full-body text searches.

Opt-In Locker Configuration

Under the Bosun storage specification, full-text search is strictly opt-in. The vault identity manifest (.bosun/locker.json) declares whether FTS indexing is active:

{
  "locker_id": "lkr_01j7m8x4k2",
  "format_version": "1.0",
  "created_at": "2026-09-14T08:00:00Z",
  "fts_enabled": false,
  "checksum": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}

When fts_enabled is false:

  1. The SQLite virtual table notes_fts is never created.
  2. Note body strings are discarded immediately after frontmatter and link extraction.
  3. The database retains only metadata: paths, titles, tags, and link edges.

Fail-Closed Query Guardrail

Attempting to execute a full-text query when FTS is disabled raises an explicit typed error rather than falling back to an unindexed LIKE '%query%' table scan across all note records:

class FtsDisabledError(RuntimeError):
    """Raised when full-text search is invoked on an unindexed locker."""
    pass
 
def search_notes(self, query: str, full_text: bool = False) -> list[NoteSearchResult]:
    if full_text:
        if not self.locker_manifest.fts_enabled:
            raise FtsDisabledError(
                f"Full-text search is disabled for locker '{self.locker_id}'. "
                "Enable 'fts_enabled: true' in .bosun/locker.json and rebuild index."
            )
        return self.projection.query_fts(query)
 
    # Fast path: prefix and keyword match on title and path
    return self.projection.query_metadata_title(query)

This prevents accidental disk-thrashing queries from degrading user interface responsiveness.

Empirical Footprint Comparison

Measurements taken across a 50,000-note vault on Linux 6.8 with 16 GB RAM:

Configuration ModeDaemon RAM (RSS)Database Disk SizeStartup Index Scan TimeTitle Search Latency (p95)
Default (Metadata Only)14.2 MB18.4 MB1.8 seconds0.8 ms
Opt-In FTS5 (Unicode61)48.6 MB162.8 MB12.4 seconds1.2 ms (Body search: 4.8 ms)
Unbounded In-Memory Tokenizer242.0 MB210.5 MB26.8 seconds1.1 ms (Body search: 3.1 ms)

By isolating full-text indexing to an opt-in toggle, Bosun PKM guarantees that the default background engine operates comfortably within a sub-20 MB memory envelope.

  • Directus Target: bosun-pkm
  • Garden Source Reference: MOC - Bosun PKM Tools, MOC - Bosun PKM Engine, MOC - The Plain-Text Longevity Standard, [BSN-1005 - Compiling the Engine to WebAssembly](BSN-1005 - Compiling the Engine to WebAssembly)

Graph View