DAT-1005 - Deterministic Lossless Conversion

Living Document Notice
Published 2026-09-14. The evolving architecture, revisions, and connected notes for this dispatch live in the Stax Digital Garden.

Deterministic Lossless Conversion

Summary

Building bidirectional AST pipelines with round-trip test suites to verify zero semantic data loss.

This technical dispatch explores the underlying architecture, data structures, and concrete implementation boundaries required for local-first data sovereignty.

The Failure Modes of String-Based Parsers

Most documentation migration scripts rely on regular expressions or linear text replacers to translate proprietary JSON document blocks into Markdown. A script identifies a heading block, prepends ## , identifies bold text spans, and wraps the content in double asterisks.

This naive approach breaks down immediately on complex document hierarchies. Real-world documents contain nested inline formatting, math equations, code blocks with custom syntax highlight keys, task lists with completion timestamps, and embedded media alignments. When regular expressions parse overlapping markup spans—such as italic text crossing over a hyperlink boundary—they generate invalid CommonMark tokens:

<!-- Malformed regex output -->
*This is [italicized and linked*](https://example.com) text.

Standard Markdown renderers fail to parse this token string deterministically, resulting in broken HTML output and lost links. Lossless data migration requires representing the document as an Abstract Syntax Tree (AST), performing tree transformations, and serializing back to target specifications with mathematical precision.

The Abstract Syntax Tree Pipeline

The FreeMyData parser pipeline operates across three isolated phases:

[Vendor JSON AST] ──> [Intermediate Unified AST] ──> [CommonMark / GFM AST]
                             │
                             ├── Validate Invariant Attributes
                             └── Inject Fallback Preservation Pragmas

1. Ingestion & Normalization: The proprietary vendor tree is deserialized into an intermediate representation where nodes conform to unified token interfaces: text, block, container, and leaf.
2. Tree Transformation: Nodes undergo structural normalization. Sibling text spans with identical formatting attributes are merged into atomic spans, preventing redundant delimiters (**foo****bar** becomes **foobar**).
3. Pragma Injection: Vendor-specific properties with no direct CommonMark equivalent—such as custom callout icon types, background highlight colors, or column layout widths—are converted into non-destructive HTML comment pragmas.

<!-- data-vendor-callout: {"type": "warning", "icon": "shield-alert"} -->
> This warning contains preserved vendor attributes that standard Markdown ignores.

When an external engine renders this Markdown document, standard readers display an ordinary blockquote. When FreeMyData or an advanced knowledge tool re-parses the document, it inspects the pragma block to restore original UI properties.

Invariant Testing and Round-Trip Validation

To guarantee that an AST conversion pipeline does not drop data, the pipeline must pass strict round-trip verification tests. If a document passes from format A to format B and back to format A, the semantic content must remain identical.

Node Category	CommonMark Element	Vendor Pragma Retention	Lossless Verification
Math Equations	$$ block delimiters	Preserves original LaTeX macro strings	Exact string match on AST leaf
Fenced Code	```lang	Preserves execution flags, file name attributes	Hash match on body content
Tabular Data	GFM Table syntax	Preserves cell alignments, merged spans	Cell coordinates match row/col index
Task Items	- [ ] checkboxes	Preserves completion author, Unix timestamp	Timestamp verified in pragma attributes

Here is an example of an AST transform pass implemented in Rust, normalizing a text leaf while maintaining attribute parity:

#[derive(Debug, PartialEq, Serialize, Deserialize)]
pub struct UnifiedNode {
    pub kind: NodeKind,
    pub children: Vec<UnifiedNode>,
    pub attributes: HashMap<String, Value>,
}
 
pub fn normalize_text_nodes(nodes: Vec<UnifiedNode>) -> Vec<UnifiedNode> {
    let mut normalized: Vec<UnifiedNode> = Vec::with_capacity(nodes.len());
 
    for current in nodes {
        if let Some(last) = normalized.last_mut() {
            if last.kind == NodeKind::Text 
                && current.kind == NodeKind::Text 
                && last.attributes == current.attributes 
            {
                if let (Some(last_text), Some(curr_text)) = (
                    last.attributes.get_mut("content"),
                    current.attributes.get("content")
                ) {
                    if let (Value::String(l), Value::String(c)) = (last_text, curr_text) {
                        l.push_str(c);
                        continue;
                    }
                }
            }
        }
        normalized.push(current);
    }
    normalized
}
 
#[test]
fn verify_roundtrip_fidelity() {
    let input = load_fixture("vendor_block_dump.json");
    let intermediate = parse_vendor_ast(&input);
    let serialized = to_commonmark(&intermediate);
    let roundtrip = parse_commonmark(&serialized);
    assert_eq!(intermediate.content_hash(), roundtrip.content_hash());
}

---

• Directus Target: freemydata
• Garden Source Reference: DAT-1002 - The Anatomy of Hostile Schemas, DAT-1004 - Flat Markdown vs Embedded SQLite, MOC - Data Liberation Workbenches, MOC - The Plain-Text Longevity Standard, MOC - Bosun PKM Tools