Stax Digital Garden

DAT-1003 - The Export Illusion

4 min read

Living Document Notice
Published 2026-09-12. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

The Export Illusion

The Export Illusion: Electric lime P1 vector CRT macro showing labyrinthine export maze with high-intensity laser vector escape beam

Summary

Why GDPR Article 20 dumps deliver hostile, nested JSON arrays designed for audit compliance rather than functional migration.

This technical dispatch explores the underlying architecture, data structures, and concrete implementation boundaries required for local-first data sovereignty.

Compliance Formats vs Ingestion Realities

Cloud service providers routinely offer export tools under account settings labeled “Download Your Archive.” While these archives satisfy statutory requirements for data portability, their internal architecture reflects the underlying storage engine of the vendor rather than an interchangeable exchange standard.

A standard export archive typically packages data into nested zip files containing multi-megabyte JSON files. Instead of providing normalized tables or flat document sequences, the records mirror internal document-database topologies. Relational entities are split across disparate files where IDs reference internal UUIDs with no schema dictionary provided. Additionally, binary media attachments—such as inline images, voice memos, and PDF attachments—are frequently omitted from the archive payload entirely, replaced instead with short-lived S3 pre-signed URLs that expire within 15 to 60 minutes of archive generation.

export_archive.zip
├── account_meta.json
├── blocks_part001.json       # Fragmented block tree (depth > 12)
├── blocks_part002.json
├── media_references.json     # AWS S3 URLs with Expires=1773345600
└── user_graph_tombstones.json # Dropped foreign key mappings

If an automated pipeline does not immediately fetch external assets, the liberated text records point permanently to dead HTTP 403 endpoints.

Shredded Block Trees and Dangling References

Consider the block storage paradigm popular in modern collaborative document editors. A document is not stored as a serialized string of Markdown or HTML; it is modeled as an acyclic directed graph of micro-blocks, each tracking indentation levels, permission masks, and revision vectors:

{
  "id": "8f3b6c2a-9e12-4d56-a789-0123456789ab",
  "parent_id": "1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
  "type": "text_block",
  "properties": {
    "title": [
      ["Document paragraph with "],
      ["embedded link", [ ["a", "https://example.com"] ]]
    ],
    "author_ref": "usr_990184"
  },
  "content_refs": [
    "c4d5e6f7-a8b9-0c1d-2e3f-4a5b6c7d8e9f"
  ]
}

When exported, the vendor serializes these blocks in arbitrary chunked batches. A child block in blocks_part002.json references a parent defined in blocks_part001.json. If a block was deleted during editing, its UUID remains referenced in the parent array, creating dangling pointer exceptions during tree traversal.

The table below outlines common structural defects found in raw vendor export payloads:

Payload Failure ModeVendor Source PatternEngineering Mitigation
Ephemeral Media LinksPre-signed S3/GCS URLs with query param expirationImmediate asynchronous asset mirroring to local storage
Cyclic Block PointersUnvalidated revision history branches in JSON treesCycle detection during topological graph sorting
Polymorphic Array DumpsArbitrary mix of objects, strings, and integers in arraysExplicit schema validation schemas using typed AST parsers
Dropped Text SpansMulti-dimensional array markup for bold/italic tokensSpan-flattening string reducers with position offsets

Reconstructing Linear Hierarchies with jq and SQLite

To convert shredded JSON trees into flat relational structures suitable for long-term storage, the pipeline must first stage raw blocks into an embedded SQLite instance. Attempting in-memory reassembly using standard scripting objects runs into heap exhaustion on archives containing more than 50,000 blocks.

First, extract and stream the raw JSON blocks directly into a staging database using a shell pipeline with jq and SQLite:

# Stream JSON array items directly into newline-delimited records
jq -c '.blocks[]' raw_export/blocks_part*.json > staging_blocks.jsonl
 
# Ingest into SQLite staging table with generated columns
sqlite3 archive_staging.db << 'EOF'
CREATE TABLE raw_blocks (
    id TEXT PRIMARY KEY,
    parent_id TEXT,
    block_type TEXT,
    content_json TEXT,
    depth INTEGER DEFAULT 0
);
 
.mode json
.import staging_blocks.jsonl raw_blocks
EOF

Once staged, a recursive common table expression (CTE) rebuilds document hierarchy and calculates absolute document ordering:

WITH RECURSIVE document_tree AS (
    -- Anchor member: root nodes without parents
    SELECT 
        id, 
        parent_id, 
        block_type, 
        content_json, 
        0 AS level,
        printf('%05d', row_number() OVER (ORDER BY id)) AS path_sort
    FROM raw_blocks
    WHERE parent_id IS NULL OR parent_id = ''
 
    UNION ALL
 
    -- Recursive member: resolve children to parents
    SELECT 
        child.id, 
        child.parent_id, 
        child.block_type, 
        child.content_json, 
        dt.level + 1,
        dt.path_sort || '/' || printf('%05d', row_number() OVER (PARTITION BY child.parent_id ORDER BY child.id))
    FROM raw_blocks child
    JOIN document_tree dt ON child.parent_id = dt.id
)
SELECT id, level, path_sort, block_type 
FROM document_tree 
ORDER BY path_sort ASC;
  • Directus Target: freemydata
  • Garden Source Reference: DAT-1002 - The Anatomy of Hostile Schemas, [BSN-1001 - Incremental Parsing with Tree-sitter](BSN-1001 - Incremental Parsing with Tree-sitter), MOC - Data Liberation Workbenches, MOC - The Plain-Text Longevity Standard, MOC - Bosun PKM Tools, [BSN-1002 - Deterministic Round-Trip Serialization](BSN-1002 - Deterministic Round-Trip Serialization)

Graph View