Stax Digital Garden

EMB-1009 - Offline Media Ingestion and Direct POSIX Synchronization

2 min read

Living Document Notice
Published 2026-09-18. The evolving architecture and revisions for this dispatch live in the Stax Digital Garden.

Offline Media Ingestion and Direct POSIX Synchronization

Offline Media Ingestion and Direct POSIX Synchronization: Warm sepia-white phosphor P4/P20 vector CRT macro showing direct offline data transmission tracks exchanging packets between local storage frames

Family media ingestion should not depend exclusively on active web browser uploads. When importing hundreds of gigabytes from DSLR SD cards or legacy hard drives, web-based multipart uploads stall, time out, and exhaust browser memory buffers.

Embers supports direct POSIX filesystem synchronization. Operators can mount external storage volumes, rsync raw media files into directory trees, and trigger background indexing passes that extract metadata and generate thumbnails without transferring bytes over browser HTTP connections.

Directory Ingestion Architecture

Direct disk ingestion bypasses network bottlenecks:

[ SD Card / External Drive ] ──► rsync -avP /mnt/card/ /var/lib/embers/library/
                                         │
                                         ▼
                             [ POSIX Media Directory ]
                                         │
                                         ▼
                            [ Embers Indexing Worker ]
                               - Inode discovery
                               - EXIF metadata extraction
                               - Sharp thumbnail creation
Ingestion MethodWeb Multipart UploadDirect POSIX Synchronization
Throughput LimitBrowser connection limits (~30 MB/s)Bus / NVMe wire speed (500+ MB/s)
ResumabilityUnreliable on browser restartNative rsync block-level delta resumption
Memory PressureHigh browser heap allocationMinimal streaming buffer overhead

Inode Scanning and Hash Verification

import fs from "fs";
import path from "path";
import crypto from "crypto";
 
export function scanDirectory(dir: string): string[] {
  const results: string[] = [];
  const entries = fs.readdirSync(dir, { withFileTypes: true });
 
  for (const entry of entries) {
    const fullPath = path.join(dir, entry.name);
    if (entry.isDirectory()) {
      results.push(...scanDirectory(fullPath));
    } else if (/\.(jpe?g|png|mp4|mov)$/i.test(entry.name)) {
      results.push(fullPath);
    }
  return results;
}

Ingestion CLI Commands

Trigger local library synchronization from the terminal:

# Synchronize local camera dump directly into Embers library
rsync -av --progress /mnt/sdcard/DCIM/ /var/lib/embers/library/imports/2026-09-18/
 
# Trigger background metadata sweep
curl -s -X POST http://127.0.0.1:3001/api/internal/ingest/rescan \
  -H "Authorization: Bearer ${EMBERS_ADMIN_TOKEN}"
  • Directus Target: embers
  • Garden Source Reference: MOC - The Digital Necropolis and Cold Decadal Storage, MOC - Bosun PKM Tools
  • Garden Source Reference: [EMB-1009 - Offline Media Ingestion and Direct POSIX Synchronization](EMB-1009 - Offline Media Ingestion and Direct POSIX Synchronization)

Graph View