Stax Digital Garden

EMB-1005 - Ingest Pipelines - Background Thumbnail Generation with Sharp

2 min read

Living Document Notice
Published 2026-09-14. The evolving architecture and revisions for this dispatch live in the Stax Digital Garden.

Ingest Pipelines - Background Thumbnail Generation with Sharp

Ingest Pipelines - Background Thumbnail Generation with Sharp: Warm sepia-white phosphor P4/P20 vector CRT macro showing image downscaling pipeline stepping into a grid matrix of thumbnail wireframes

Displaying high-resolution camera raw files directly on mobile browsers causes bandwidth exhaustion and interface stutter. Embers structures media ingestion around an asynchronous Sharp worker queue that generates deterministic WebP thumbnails and strips tracking metadata before committing assets.

Thumbnail Pipeline Constraints

High-resolution camera sensors capture images exceeding 48 megapixels with file sizes upwards of 25 megabytes. Delivering full-resolution originals to gallery views degrades client rendering performance and saturates household upstream internet connections.

Embers implements a three-tier thumbnail generation pipeline using native libvips via Sharp. As uploads arrive, background workers generate compressed WebP variants while preserving the pristine original in cold storage:

[Raw Media Upload: 25 MB original]
                 |
                 v
      [Sharp libvips Pipeline]
                 |
                 +---> Thumbnail (160x160 WebP, 6 KB)
                 +---> Preview   (1280x720 WebP, 85 KB)
                 +---> Display   (2560x1440 WebP, 340 KB)

Asynchronous Worker Queue Implementation

Thumbnail workers run on separate thread pools to avoid blocking the primary HTTP event loop:

// src/workers/thumbnailWorker.ts
import sharp from 'sharp';
import fs from 'fs';
import path from 'path';
 
export interface ThumbnailJob {
  sourcePath: string;
  targetDir: string;
  mediaId: string;
}
 
export async function processThumbnail(job: ThumbnailJob): Promise<void> {
  const pipeline = sharp(job.sourcePath).rotate();
 
  // Strip sensitive GPS and tracking EXIF from generated web derivatives
  await Promise.all([
    pipeline
      .clone()
      .resize(160, 160, { fit: 'cover' })
      .webp({ quality: 80 })
      .toFile(path.join(job.targetDir, `${job.mediaId}_thumb.webp`)),
 
    pipeline
      .clone()
      .resize(1280, 1280, { fit: 'inside', withoutEnlargement: true })
      .webp({ quality: 82 })
      .toFile(path.join(job.targetDir, `${job.mediaId}_preview.webp`))
  ]);
}

Batch Processing Telemetry

Inspect thumbnail generation throughput and storage compression rates:

# Benchmark thumbnail generation across 100 sample images
$ embers-worker-benchmark --input /mnt/test-raw --count 100
Processing 100 raw images (2.4 GB total)...
Completed in 3.42s (29.2 images/sec)
Total derived storage: 12.8 MB (99.4% reduction from original)
Worker thread RSS: 64.2 MB
  • Directus Target: embers
  • Garden Source Reference: MOC - The Digital Necropolis and Cold Decadal Storage, MOC - Bosun PKM Tools
  • Garden Source Reference: ingest-pipelines-background-thumbnail-generation-with-sharp, sharp-libvips, image-optimization, embers-ingest

Graph View