Stax Digital Garden

EMB-1003 - Pluggable Storage Drivers for Local Disk and Private S3

3 min read

Living Document Notice
Published 2026-09-12. The evolving architecture and revisions for this dispatch live in the Stax Digital Garden.

Pluggable Storage Drivers for Local Disk and Private S3

Pluggable Storage Drivers for Local Disk and Private S3: Warm sepia-white phosphor P4/P20 vector CRT macro showing bilateral storage drivers: local disk platter sectors on left and private object store cluster on right

Family photo collections vary from small SSD arrays attached to a home server to multi-terabyte cold archives hosted on dedicated MinIO instances. Embers separates media storage behind a unified driver interface supporting atomic POSIX writes and streaming S3 object uploads without application-level branching.

Decoupling Media Assets from Database State

Monolithic media servers tie asset storage directly to proprietary database BLOB fields or hardcoded local directory paths. This design complicates storage migration when disk volumes fill up.

Embers routes all binary I/O through a driver contract. The database stores only media UUIDs, content hashes, and dimensions. Binary bytes flow through pluggable storage providers:

[Ingest Controller]
         |
         v
[StorageDriver Interface]
         |
         +---> LocalDiskDriver (Direct ext4/zfs POSIX atomic write)
         |
         +---> S3StorageDriver (MinIO / AWS S3 streaming multipart)

Storage Driver Contract Implementation

The storage interface standardizes streaming writes, range reading, and content deduplication:

// src/storage/driver.ts
import { Readable } from 'stream';
import fs from 'fs';
import path from 'path';
 
export interface StorageDriver {
  write(key: string, stream: Readable): Promise<{ bytesWritten: number; sha256: string }>;
  read(key: string, start?: number, end?: number): Promise<Readable>;
  delete(key: string): Promise<void>;
  exists(key: string): Promise<boolean>;
}
 
export class LocalDiskDriver implements StorageDriver {
  constructor(private basePath: string) {}
 
  async write(key: string, stream: Readable): Promise<{ bytesWritten: number; sha256: string }> {
    const targetPath = path.join(this.basePath, key);
    await fs.promises.mkdir(path.dirname(targetPath), { recursive: true });
    const outStream = fs.createWriteStream(targetPath, { flags: 'wx' });
 
    return new Promise((resolve, reject) => {
      stream.pipe(outStream);
      outStream.on('finish', () => resolve({ bytesWritten: outStream.bytesWritten, sha256: key }));
      outStream.on('error', reject);
    });
  }
 
  async read(key: string, start?: number, end?: number): Promise<Readable> {
    const targetPath = path.join(this.basePath, key);
    return fs.createReadStream(targetPath, { start, end });
  }
 
  async delete(key: string): Promise<void> {
    await fs.promises.unlink(path.join(this.basePath, key));
  }
 
  async exists(key: string): Promise<boolean> {
    try {
      await fs.promises.access(path.join(this.basePath, key));
      return true;
    } catch {
      return false;
    }

Storage Engine Configuration Schema

Operators switch storage targets by updating the configuration environment block:

# Verify active storage backend status
$ cat /etc/embers/config.env
EMBERS_STORAGE_DRIVER=local
EMBERS_LOCAL_STORAGE_PATH=/mnt/storage/embers/media
EMBERS_STORAGE_QUOTA_GB=2000
 
# Test storage driver connectivity and read latency
$ embers-cli test-storage
Backend: LocalDiskDriver (/mnt/storage/embers/media)
Read test: 4.2 MB read in 3.1ms (1354 MB/s)
Write test: 10.0 MB write in 8.4ms (1190 MB/s)
Storage status: NOMINAL
  • Directus Target: embers
  • Garden Source Reference: MOC - The Digital Necropolis and Cold Decadal Storage, MOC - Bosun PKM Tools
  • Garden Source Reference: pluggable-storage-drivers-for-local-disk-and-private-s3, storage-architecture, minio-integration, embers-storage

Graph View