The export that eats the heap

The problem

You call an export endpoint. The vendor hands back one JSON array with tens of thousands of rows. You await it, parse it, iterate. Then the catalog grows and the process dies.

Parsing costs several times the wire size, because JSON.parse holds the whole string and builds the object tree. A documented sync of 22,000 products in an 84 MB response exhausted ~2.1 GB on a 4 GB VPS; the streaming rewrite ran at 180 MB peak.

This failure mode is unlike every other scenario here. Nothing returns wrong data — the process just dies, usually after a dataset crossed a threshold nobody was watching.

The common solutions

What StitchAPI does

It ships a genuinely O(1) NDJSON decoder. Driven directly, decode: 'ndjson' held 0.8 MB of retained heap for 1,000,000 rows and 214 MB of wire, moving less than 15% across a 1000× change in workload. That is the real thing.

And the engine retains every chunk one line later. Through runStreaming the same decoder is linear — 3.5 MB → 30.2 MB from 10k to 100k rows — because every delta is pushed onto an accumulator so the terminal result can mirror the whole spine.

So the seam that does work is Surface.stream — not to replace the decoding, which is already O(1), but because the engine retains whatever that hook yields. Yield one small receipt per batch instead of one row per row:

const rows = stitch({
    url: 'https://api.vendor.com/export.ndjson',
    kind: batchedExport({ size: 1_000, onBatch: writeToDb }), // yields a receipt, not rows
    stream: 'ndjson',
});

Measured: 1.3 MB retained for 100,000 rows against the buffered baseline's 53.8 MB — a 40× cut — and flat: 1.3 MB at 1,000 rows, 1.4 MB at 100,000. Under a 96 MB heap ceiling, 400,000 rows that killed the buffered path outright processed completely at 1.3 MB.

Fairness on the baseline: the buffered multiplier measured 2.5×, and it is JSON.parse's, not the library's — StitchAPI's overhead over a bare JSON.parse of the same bytes was 0.2 MB. It adds no copy. It also removes none.

decode: 'json' on one array: right answer — and, since #665, right memory

The structural decoder's emission was always correct, and impressively so — one delta per element, holding up under , ] } inside string values, escaped quotes, embedded newlines, pretty-printed multi-line records, deep nesting, and 1-character chunk boundaries.

Its memory was not. The decoder retained the whole array text at ~0.9× the wire in quadratic time, and tripped its own default cap mid-export: a 60,000-row array delivered 37,312 rows and then error/done(ok: false) under "a malformed or never-closing value was streamed" — which the vendor did not do. That defect was measured here, filed as #659 §2, and fixed by #665: a top-level array records no compaction floor of its own, so emitted elements are released as they close.

What the fix does not change: through the engine, decode: 'json' now costs exactly what ndjson costs — 30.2 MB at 100,000 rows either way — because the accumulator above it still retains every element. The decoder stopped buffering; the engine hasn't. What it does change: the batched seam above no longer depends on being offered NDJSON — over one 100,000-row array the same surface runs at 1.4 MB, where it used to pay 19 MB for the decoder's buffer sitting upstream of the seam.

What StitchAPI does not solve here

1. No memory-bounded path is reachable from config alone. Every decoder × every accessor is O(N) through the engine.
2. No size guard, threshold, warning or event on the buffered path. A 21 MB response emits the same four events a 40-byte one does. When the wall arrives it is V8's: 400,000 rows under a 96 MB ceiling gave FATAL ERROR: … JavaScript heap out of memory, exit 134 — no catchable error, no error event, no finally.
3. stream.buffer.chars is a malformed-input guard, not a budget. It bounds one line or one un-closed value — 20,000 well-formed rows streamed cleanly through a 1,000-character cap while the engine accumulated all 20,000. On a buffered stitch it typechecks and is inert.
4. pick and transform are silently inert on a stream — transform called zero times over 200 deltas. No info, no drift finding, no throw. A stitch that carries them and is later switched to kind: stream keeps compiling and quietly stops reshaping.
5. output on a stream validates without transforming. The buffered path serves the validated value; the streaming path keeps only the errors. The same coercing schema reshapes your data on await and silently does not on .stream().
6. A failing row is a circuit breaker, not a filter — contract violation (drift), the stream ends, and the rest is never read.
7. No batching primitive. There's no paginate-style chunking for streams; the 75 lines above are yours.

See also