NXT-1007 - Hydrating FreeNext from FreeMyData

Living Document Notice
Published 2026-09-16. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Hydrating FreeNext from FreeMyData

Summary

Migrating personal records out of cloud platforms often produces unstructured document archives, monolithic JSON blobs, or raw SQLite database dumps. While extraction utilities like the FreeMyData extraction engine unlock raw data silos directly within the browser, converting these heterogenous records into structured note collections requires an automated ingestion and normalization pipeline.

FreeNext integrates directly with FreeMyData outputs through a client-side hydration engine. Running entirely inside local Web Workers, this pipeline parses extracted database rows, maps arbitrary JSON properties into YAML frontmatter schemas, and writes individual CommonMark documents into local vault storage without intermediate server hops.

The Ingestion Pipeline Architecture

The hydration pipeline consumes extracted database streams, applies structural transformations, and serializes files directly into the user-selected filesystem directory.

+-------------------------------------------------------------+
|                     FreeMyData Output                       |
|         (Raw SQLite Tables or Chunked JSON Exports)         |
+-------------------------------------------------------------+
                               |
                               | Streaming Record Batches
                               v
+-------------------------------------------------------------+
|                 FreeNext Hydration Worker                   |
|       +---------------------------------------------+       |
|       |             Schema Field Mapper             |       |
|       +---------------------------------------------+       |
|                              |                              |
|                              v                              |
|       +---------------------------------------------+       |
|       |         Markdown Generator & AST Sanitizer  |       |
|       +---------------------------------------------+       |
+-------------------------------------------------------------+
                               |
                               | Serialized File Batches
                               v
+-------------------------------------------------------------+
|                 Local Vault Storage Directory               |
|            [notes/2026-09-01-entry.md]                      |
|            [attachments/media-001.png]                      |
+-------------------------------------------------------------+

Processing occurs in memory streams, keeping RAM usage capped under 40 megabytes even when processing archives with tens of thousands of records.

Transformation Mapping Matrix

Incoming SaaS exports vary widely in naming conventions and data structures. The hydration engine uses declarative mapping tables to normalize source fields into canonical FreeNext note properties.

Source Platform Record Field	Intermediate Extracted Type	Canonical FreeNext Field	Serialization Format
record_id / uuid	String (Hex/UUID)	frontmatter.id	Quoted string identifier
created_time / epoch_ms	Numeric Epoch	frontmatter.date	ISO 8601 Date (2026-09-16)
tags_list / categories	Comma-delimited text	frontmatter.tags	YAML array sequence
raw_html_body	HTML markup string	Markdown Body	CommonMark with standard syntax
binary_attachment_blob	Base64 string	Disk Asset File	Relative path link assets/hash.ext
is_archived	Integer flag (0 or 1)	frontmatter.archived	Boolean true or false

This mapping isolates upstream format quirks from the primary vault structure, ensuring all notes adhere to consistent organization rules.

Hydration Pipeline Implementation

The TypeScript worker implementation below illustrates batch stream processing from an extracted SQLite database cursor into individual vault files:

// hydration-worker.ts: Ingestion of FreeMyData tables
export interface IngestionConfig {
  tableName: string;
  idField: string;
  dateField: string;
  titleField: string;
  bodyField: string;
  tagField?: string;
  targetDirectoryHandle: FileSystemDirectoryHandle;
}
 
export async function hydrateVaultFromCursor(
  records: AsyncIterable<Record<string, unknown>>,
  config: IngestionConfig
): Promise<number> {
  let processedCount = 0;
 
  for await (const row of records) {
    const rawId = String(row[config.idField] || crypto.randomUUID());
    const title = String(row[config.titleField] || 'Untitled Note');
    const rawDate = row[config.dateField];
    const isoDate = normalizeDate(rawDate);
    const body = String(row[config.bodyField] || '');
 
    const tags: string[] = [];
    if (config.tagField && row[config.tagField]) {
      const rawTags = String(row[config.tagField]);
      tags.push(...rawTags.split(',').map(t => t.trim()).filter(Boolean));
    }
 
    const markdownContent = serializeNoteEnvelope({
      id: rawId,
      title,
      date: isoDate,
      tags,
      body,
    });
 
    const sanitizedFilename = sanitizeFilename(`${isoDate}-${title}.md`);
    await writeVaultFile(config.targetDirectoryHandle, sanitizedFilename, markdownContent);
 
    processedCount++;
  }
 
  return processedCount;
}
 
function normalizeDate(raw: unknown): string {
  if (typeof raw === 'number') {
    return new Date(raw).toISOString().split('T')[0];
  }
  if (typeof raw === 'string') {
    const parsed = Date.parse(raw);
    if (!isNaN(parsed)) {
      return new Date(parsed).toISOString().split('T')[0];
    }
  }
  return new Date().toISOString().split('T')[0];
}
 
function serializeNoteEnvelope(note: {
  id: string;
  title: string;
  date: string;
  tags: string[];
  body: string;
}): string {
  const yamlLines = [
    '---',
    `id: "${note.id}"`,
    `title: "${note.title.replace(/"/g, '\\"')}"`,
    `date: ${note.date}`,
  ];
 
  if (note.tags.length > 0) {
    yamlLines.push('tags:');
    for (const tag of note.tags) {
      yamlLines.push(`  - ${tag}`);
    }
  }
 
  yamlLines.push('---', '', `# ${note.title}`, '', note.body);
  return yamlLines.join('\n');
}
 
function sanitizeFilename(name: string): string {
  return name.replace(/[/\\?%*:|"<>]/g, '-').slice(0, 120);
}
 
async function writeVaultFile(
  dir: FileSystemDirectoryHandle,
  filename: string,
  content: string
): Promise<void> {
  const fileHandle = await dir.getFileHandle(filename, { create: true });
  const writable = await fileHandle.createWritable();
  await writable.write(content);
  await writable.close();
}

By connecting extraction and hydration via local pipelines, users migrate personal data out of cloud silos into structured Markdown vaults without vendor lock-in.

---

• Directus Target: freenext
• Garden Source Reference: DAT-1001 - Data Liberation Pipeline, NXT-1002 - SQLite in the Browser via OPFS, MOC - Data Liberation Workbenches, MOC - The Plain-Text Longevity Standard, MOC - Local-First Systems and Synchronization, MOC - Bosun PKM Tools