HAR-1007 - The Ingestion Pipeline from Vault to Edge

Living Document Notice
Published 2026-09-16. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

The Ingestion Pipeline from Vault to Edge

Summary

Local-first knowledge management architectures depend on local text files stored as plain markdown on the operator’s disk. However, bridging private Obsidian vaults to public web distribution requires an automated, durable ingestion pipeline that enforces publishing boundaries, sanitizes internal links, updates database records, and purges edge caches without manual intervention.

The Bosun publishing pipeline coordinates note transformations across three specialized services. Tender parses local vault notes and normalizes markdown ASTs; Harbormaster orchestrates synchronization tasks and updates the Directus content store; and Harbor immediately re-renders the updated markup while warming edge CDN points of presence. This dispatch traces the exact event sequence that moves a file from local disk to public edge delivery.

Multi-Stage Ingestion Lifecycle

The transformation pipeline converts unstructured desktop markdown into structured edge content across four sequential phases:

┌────────────────────────┐
│  Local Obsidian Vault  │ (Plain Markdown + YAML frontmatter)
└───────────┬────────────┘
            │ 1. Git Push / Local Watcher Event
            ▼
┌────────────────────────┐
│      Tender CLI        │ (AST Parsing, wikilink resolution, asset hashing)
└───────────┬────────────┘
            │ 2. Canonical JSON Payload via Internal HTTP
            ▼
┌────────────────────────┐
│  Harbormaster Control  │ (Database synchronization & revision tracking)
└───────────┬────────────┘
            │ 3. Database Upsert into Directus & Invalidation Broadcast
            ▼
┌────────────────────────┐
│   Harbor Edge Node     │ (Cache eviction, SSR pre-render, CDN cache warm)
└────────────────────────┘

1. Extraction and AST Normalization (Tender)

Tender reads the raw .md file from disk. It validates that the document frontmatter contains draft: false and a designated blog key. Tender constructs a unified abstract syntax tree (AST), transforms internal wikilinks into spec-compliant relative HTML anchors, and calculates SHA-256 hashes of all embedded image attachments.

2. Event Dispatch (Harbormaster)

Once Tender validates the document, it emits a structured ingest.note.published event to the Harbormaster KPP protocol fleet controller. Harbormaster checks document revision history, resolves tenant identity from the note’s target blog attribute, and executes an idempotent SQL upsert into the Directus articles collection.

3. Edge Notification and Cache Invalidation (Harbor)

Upon receiving the commit confirmation from Directus, Harbormaster fires an internal webhook to Harbor. Harbor evicts the document’s cached HTML from local LRU buffers and dispatches an asynchronous cache purge request to Cloudflare’s Purge API by Cache-Tag.

4. Cache Warming

Harbor immediately triggers an internal loopback fetch against the invalidated URL. By rendering the page once immediately after invalidation, the new HTML response sits pre-cached in memory before external web crawlers or subscribers arrive.

Stage	Subsystem	Inbound Payload	Outbound Artifact	Execution SLA
Parsing	Tender	Raw .md + vault attachments	Clean HTML AST + JSON metadata	< 120ms
Orchestration	Harbormaster	Tender ingest event payload	Directus relational record insert	< 250ms
Eviction	Harbor Edge	invalidate webhook (slug)	Local LRU cache purge	< 10ms
Purge Broadcast	Cloudflare API	API Token + Cache-Tag array	Edge edge POP invalidation	< 450ms
Warming	Harbor Edge	Synthetic loopback GET	Pre-warmed warm cache buffer	< 35ms

Harbormaster Invalidation Webhook Handler in Harbor

Harbor listens for the cache warm signal across the internal bridge network:

import { Hono } from "hono";
 
const app = new Hono();
 
app.post("/internal/ingest/warm", async (c) => {
  const authHeader = c.req.header("authorization");
  if (authHeader !== `Bearer ${process.env.HARBOR_INTERNAL_KEY}`) {
    return c.text("Unauthorized", 401);
  }
 
  const { slug, blog, cacheTags } = await c.req.json();
 
  if (!slug || !blog) {
    return c.json({ error: "Missing slug or blog identifier" }, 400);
  }
 
  // 1. Evict local memory cache
  localCache.delete(`/posts/${slug}`);
 
  // 2. Dispatch edge CDN purge asynchronously
  if (process.env.CLOUDFLARE_ZONE_ID && process.env.CLOUDFLARE_API_TOKEN) {
    c.executionCtx?.waitUntil(
      fetch(`https://api.cloudflare.com/client/v4/zones/${process.env.CLOUDFLARE_ZONE_ID}/purge_cache`, {
        method: "POST",
        headers: {
          "Authorization": `Bearer ${process.env.CLOUDFLARE_API_TOKEN}`,
          "Content-Type": "application/json"
        },
        body: JSON.stringify({ tags: cacheTags || [`post-${slug}`] })
      })
    );
  }
 
  // 3. Warm local SSR cache immediately
  c.executionCtx?.waitUntil(
    warmLocalRoute(`/posts/${slug}`)
  );
 
  return c.json({ status: "warming_initiated", slug });
});
 
async function warmLocalRoute(path: string) {
  try {
    const res = await fetch(`http://127.0.0.1:3000${path}`, {
      headers: { "x-harbor-warm": "1" }
    });
    console.log(`[Cache Warm] ${path} returned status ${res.status}`);
  } catch (err) {
    console.error(`[Cache Warm Failed] ${path}:`, err);
  }
}
 
export default app;

Pipeline Verification and Event Emitting Probing

Test the ingestion event pathway using curl to simulate Tender and Harbormaster webhooks:

# Verify Tender local transformation check
tender parse "02 Review/harbor/HAR-1007 - The Ingestion Pipeline from Vault to Edge.md" --strict
 
# Dispatch synthetic cache warm event to Harbor internal listener
curl -i -X POST http://127.0.0.1:3000/internal/ingest/warm \
  -H "Authorization: Bearer ${HARBOR_INTERNAL_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "slug": "the-ingestion-pipeline-from-vault-to-edge",
    "blog": "harbor",
    "cacheTags": ["post-the-ingestion-pipeline-from-vault-to-edge"]
  }'
 
# Confirm local warmed status in Harbor runtime logs
docker logs --tail 20 harbor_edge | grep "Cache Warm"

---

• Directus Target: harbor
• Garden Source Reference: [HAR-1007 - The Ingestion Pipeline from Vault to Edge](HAR-1007 - The Ingestion Pipeline from Vault to Edge), MOC - Harbor Ecosystem, MOC - Local-First Systems and Synchronization, MOC - Bosun PKM Tools, MOC - Harbormaster Protocol