Skip to content

Commit 0a62f5c

Browse files
oddharshclaude
andauthored
docs(compression): node 26 gave brotli a dictionary, so retire that limit (#305)
Gotcha 14 said `node:zlib`'s brotli has NO dictionary parameter, listed the nine params as the whole list, and rested part of the dcz-over-dcb case on dcb needing the `brotli` CLI back in the build path. Node 26 takes a `dictionary` option (nodejs/node#61763, merged 2026-02-13) and `.node-version` is already 26, so CI has it too. Measured on v26.7.0, q11 over a 21KB target: 70 bytes with no dictionary against 18 with one. The CONCLUSION is untouched, because dcz won on decode and neither encoder's availability was ever the axis. What changed is one of the stated reasons. Four copies of the claim moved together, per gotcha 18's own lesson about grepping the tree when a note lands: CLAUDE.md twice, the build.mjs comment at the delta emitter, the roll-shell-dictionary.mjs header, and /garage/compression, which carried it in prose AND in an understanding-check explanation. While in that page, its dcz-vs-dcb table still held the pre-correction figures (79 vs 80 bytes, "about twice as fast"). The 2026-07-28 re-measure against real dictionaries never reached it: dcb is 245 bytes ahead across all 12 pairs (6.8%, winning 11 of 12) and dcz decodes 8.3x faster, not 2x. Both inputs to the original argument were wrong, in opposite directions, and the call survived on the structural point instead: decode scales with the RECONSTRUCTION, not the delta, so a smaller dcb delta never shrinks the decode gap. The quiz's correct answer restated the old figures verbatim and now states that. Adds one general rule, since three runtimes disagree on this single option: probe dictionary support, never infer it. node 26 honours it, workerd accepts it for zstd and silently ignores it, and bun does the same through 1.3.14 with the fix landing in oven-sh/bun#34427 (unreleased; verified on 1.4.0-canary.1). The failure is silent every time, because a frame compressed WITHOUT the dictionary still decodes fine WITH it, so the only signal is a byte count that never shrank. Co-authored-by: Aadharsh Pannirselvam <19518661+oddharsh@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent e0e447a commit 0a62f5c

4 files changed

Lines changed: 73 additions & 26 deletions

File tree

‎CLAUDE.md‎

Lines changed: 31 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1505,12 +1505,34 @@ npm run deploy
15051505
not one on these. Do not re-open it on the size table alone: that table now
15061506
favours dcb, and it is the wrong axis.
15071507

1508-
One cost the byte comparison hides: `node:zlib`'s brotli has NO dictionary
1509-
parameter (MODE, QUALITY, LGWIN, LGBLOCK, DISABLE_LITERAL_CONTEXT_MODELING,
1510-
SIZE_HINT, LARGE_WINDOW, NPOSTFIX, NDIRECT is the whole list). Those dcb
1511-
figures came from shelling out to the `brotli` CLI with `-D`. Switching would
1512-
reintroduce a CLI dependency in the build path, which is precisely what moving
1513-
the deltas into the build deleted.
1508+
One cost the byte comparison used to hide is RETIRED, and it is worth reading
1509+
as a correction rather than a fact. This note said `node:zlib`'s brotli has NO
1510+
dictionary parameter, listed the nine params as the whole list, and concluded
1511+
that switching would reintroduce a `brotli` CLI dependency in the build path,
1512+
which is precisely what moving the deltas into the build deleted. Node 26 takes
1513+
a `dictionary` option (nodejs/node#61763, merged 2026-02-13), and
1514+
`.node-version` is 26, so CI has it too. Measured 2026-08-10 on node v26.7.0,
1515+
q11 over a 21KB target: 70 bytes with no dictionary against 18 with one. The
1516+
dcb figures above still came from the CLI with `-D`, because that was the only
1517+
door on the day they were taken.
1518+
1519+
The CLI-dependency argument is therefore gone while the CONCLUSION is
1520+
untouched, since dcz won on DECODE and neither engine's encoder was ever the
1521+
question. Re-open it only with a fresh decode measurement, never on the size
1522+
table (see above) and no longer on this paragraph.
1523+
1524+
**Probe an engine's dictionary support, never infer it.** Three runtimes
1525+
disagree on this one option. Node 26 honours it. workerd accepts it for zstd
1526+
and silently ignores it, measured 2026-08-05, which is why `/terminal` ships no
1527+
delta calculator and says so on the page. Bun does the same thing through
1528+
1.3.14, its current stable, with the fix landing 2026-07-18 in oven-sh/bun#34427
1529+
and verified here on `1.4.0-canary.1`. **The failure is silent in every case,
1530+
because a frame compressed WITHOUT the dictionary still decodes fine WITH it**,
1531+
so nothing throws and the only signal is a byte count that never shrank. The
1532+
control is four lines: compress one target with no dictionary, the right one,
1533+
and a wrong one. An engine that honours the option prints a smaller number for
1534+
the right dictionary alone, and an engine that ignores it prints the same
1535+
number three times.
15141536

15151537
**zstd above level 19 is dead weight here.** Levels 19, 20, 21, 22, 22+long-
15161538
distance-matching and 22+btultra2 produce BYTE-IDENTICAL output on all 9 shell
@@ -1527,8 +1549,9 @@ npm run deploy
15271549

15281550
Deltas are BUILD OUTPUT, generated by build.mjs with `node:zlib`'s zstd. An
15291551
earlier version of this note said dictionary compression was "unreachable from
1530-
Node" and shipped a workstation script with committed artifacts; that limit is
1531-
BROTLI's, and generalizing it to zstd was wrong. `zstdCompressSync` takes a
1552+
Node" and shipped a workstation script with committed artifacts; that limit was
1553+
BROTLI's, generalizing it to zstd was wrong, and node 26 has since retired the
1554+
brotli half of it too (see above). `zstdCompressSync` takes a
15321555
`dictionary` option, it beats shelling out (116 bytes where the zstd CLI gave
15331556
120), and the foreign `zstd -d -D` CLI decodes Node's output byte-exact,
15341557
skippable prefix included — the interop check that matters, since the real

‎build.mjs‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1595,8 +1595,10 @@ for (const [file, srcPath, marker] of SHELLS) {
15951595
// luna.css change: 116 bytes against 7,615.
15961596
//
15971597
// This used to be a workstation script with committed artifacts, on the belief that
1598-
// dictionary compression was unreachable from Node. That was true of BROTLI and I wrongly
1599-
// generalized it: node:zlib's zstd DOES take a `dictionary` option. It is also better
1598+
// dictionary compression was unreachable from Node. That was true of BROTLI AT THE TIME and I
1599+
// wrongly generalized it: node:zlib's zstd DOES take a `dictionary` option. (node 26 has since
1600+
// given brotli one too, nodejs/node#61763, which retires the limit rather than the choice —
1601+
// dcz won on decode speed, not on either encoder's availability. See gotcha 14.) It is also better
16001602
// than shelling out — 116 bytes where the zstd CLI produced 120 — and portable, verified
16011603
// by having the foreign `zstd -d -D` CLI decode Node's bytes byte-exact, skippable prefix
16021604
// and all. That interop check is the one that matters, because the real decoder is a

‎holding/garage/compression.html‎

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -379,24 +379,36 @@ <h2>Deltas, and why they ship as dcz</h2>
379379
</p>
380380
<p>
381381
RFC 9842 defines two encodings for this, and Cloudflare passes both through identically, so
382-
the choice is free and purely about engineering. Brotli is nominally smaller. zstd decodes
383-
about twice as fast:
382+
the choice is free and purely about engineering. Brotli compresses smaller. zstd decodes
383+
much faster:
384384
</p>
385385
<table class="cmp">
386386
<thead><tr><th>measure</th><th>dcb (brotli)</th><th>dcz (zstd)</th></tr></thead>
387387
<tbody>
388-
<tr><td>the real luna.css delta</td><td>79 bytes</td><td>80 bytes</td></tr>
389-
<tr><td>source-level deltas, larger</td><td class="y">5-8% smaller</td><td>n/a</td></tr>
390-
<tr><td>decode, bare 46KB asset</td><td>0.094 ms</td><td class="y">0.046 ms</td></tr>
391-
<tr><td>throughput</td><td>471 MB/s</td><td class="y">954 MB/s</td></tr>
388+
<tr><td>all 12 shipping deltas, total</td><td class="y">3,344 bytes</td><td>3,589 bytes</td></tr>
389+
<tr><td>widest single gap (nav.js)</td><td class="y">685 bytes</td><td>729 bytes</td></tr>
390+
<tr><td>decode, 47,615 B rebuilt</td><td>0.1368 ms</td><td class="y">0.0165 ms</td></tr>
391+
<tr><td>throughput</td><td>332 MiB/s</td><td class="y">2,752 MiB/s</td></tr>
392392
</tbody>
393393
</table>
394394
<p>
395-
I argued for brotli on bytes and got talked out of it, correctly. Bytes are a proxy;
396-
latency is the thing you actually want. And the byte case collapses on inspection: on the
397-
pair this site really ships, brotli wins by <em>one byte</em>. The 5-8% edge only appears
398-
on much larger deltas than a shell asset produces. Meanwhile the decode numbers came off a
399-
fast laptop, so the gap widens on the phones that need it most.
395+
I argued for brotli on bytes and got talked out of it, correctly, and then the numbers I
396+
argued from turned out to be wrong in <em>both</em> directions. Re-measured against real
397+
dictionaries rather than stand-ins, brotli's size edge is bigger than the one byte I had
398+
(245 bytes across the set, 6.8%, winning 11 of 12), and zstd's decode edge is far bigger
399+
than the 2x I had (8.3x). The call held anyway, which is the interesting part.
400+
</p>
401+
<p>
402+
It held for a structural reason: <strong>decode scales with the reconstruction, not with
403+
the delta.</strong> A 685-byte dcb delta still rebuilds 47,615 bytes, so making the delta
404+
smaller never makes the decode gap smaller. Break-even at 9 Mbps is about 135 bytes saved
405+
per asset and dcb saves 44, so dcz wins by roughly 3x. On a slow phone the margin widens,
406+
because decode scales with the CPU while 44 bytes stays 44 bytes.
407+
</p>
408+
<p>
409+
Worth saying plainly: on the old numbers this was a coin flip, and on these it is not. The
410+
size table now favours brotli by more than it ever did, and that table is the wrong axis to
411+
decide on.
400412
</p>
401413
<p>
402414
dcz's framing is the tidier of the two as a bonus. The dictionary hash rides in a Zstandard
@@ -422,7 +434,7 @@ <h2>Deltas, and why they ship as dcz</h2>
422434
<h2>One more over-generalization</h2>
423435
<p>
424436
Dictionary compression is unreachable from Node's zlib, so the deltas started life as a
425-
workstation script with committed artifacts. That sentence is true of brotli. I applied it
437+
workstation script with committed artifacts. That sentence was true of brotli. I applied it
426438
to zstd without checking, and <code>zstdCompressSync</code> takes a <code>dictionary</code>
427439
option. It also beats shelling out to the CLI: 116 bytes against 120.
428440
</p>
@@ -433,6 +445,15 @@ <h2>One more over-generalization</h2>
433445
interop: the foreign <code>zstd -d -D</code> CLI decodes Node's output byte-exact, prefix
434446
included, because the real decoder is a browser and not Node.
435447
</p>
448+
<p>
449+
<strong>Update, August 2026:</strong> the brotli half expired too. Node 26 added a
450+
<code>dictionary</code> option to brotli
451+
(<a href="https://github.com/nodejs/node/pull/61763" rel="external">nodejs/node#61763</a>),
452+
so on v26.7.0 a quality-11 compress of a 21 KB target goes from 70 bytes with no dictionary
453+
to 18 with one. The limit I over-generalized was accurate about brotli the day I wrote it
454+
and is now accurate about neither encoder. So a runtime limit has a shelf life, and the
455+
version you measured on belongs in the note next to the measurement.
456+
</p>
436457

437458
<h2>What shipped</h2>
438459
<table class="cmp">
@@ -511,17 +532,17 @@ <h2>The part that transfers</h2>
511532
"q": "The deltas ship as dcz even though brotli measured smaller. Why?",
512533
"options": [
513534
{
514-
"t": "zstd decodes about 2x faster, and on the pair this site actually ships brotli's advantage was one byte.",
535+
"t": "Decode scales with the reconstruction rather than the delta, so brotli's smaller delta never shrinks zstd's 8.3x decode lead.",
515536
"ok": true,
516-
"why": "Bytes are a proxy for latency. The 5-8% brotli edge only shows up on deltas far larger than a shell asset produces, while the decode gap grows on slow CPUs."
537+
"why": "Re-measured against real dictionaries, dcb is 245 bytes (6.8%) ahead across all 12 pairs, and dcz still wins: rebuilding 47,615 bytes costs 0.0165 ms against 0.1368. Break-even at 9 Mbps needs ~135 bytes saved per asset and dcb saves 44."
517538
},
518539
{
519540
"t": "Cloudflare only passes dcz through; dcb is not supported at the edge.",
520541
"why": "Both are passed through identically, on all plans. That is exactly why the choice was free to make on engineering grounds."
521542
},
522543
{
523544
"t": "Because brotli cannot use a custom dictionary, so dcb was never actually available.",
524-
"why": "Brotli does custom dictionaries fine via its CLI. What it lacks is a dictionary option in Node's zlib, which is a tooling gap rather than a format limit."
545+
"why": "Brotli does custom dictionaries fine. What it lacked was a dictionary option in Node's zlib, which was a tooling gap rather than a format limit, and node 26 closed even that."
525546
}
526547
]
527548
},

‎scripts/roll-shell-dictionary.mjs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@
55
//
66
// This is all that remains of the old gen-shell-deltas.mjs. The deltas themselves moved
77
// into build.mjs once it turned out node:zlib's zstd takes a `dictionary` option (the
8-
// "unreachable from Node" limit was BROTLI's, and I had over-generalized it), so they are
8+
// "unreachable from Node" limit was BROTLI's at the time, I had over-generalized it, and
9+
// node 26 has since given brotli one as well), so they are
910
// build output now: no zstd CLI, no committed artifacts, nothing to forget.
1011
//
1112
// Rolling stays a deliberate human step for one reason: it WRITES INTO THE SOURCE TREE, and

0 commit comments

Comments
 (0)