---
title: "aadhar.sh/garage/content-addressed chunking"
description: "Live FastCDC chunking in the browser: why a one-byte edit re-sends the whole file under fixed chunks and almost nothing under content-defined ones."
path: "/garage/chunks"
section: "garage"
kind: "content"
updated: "2026-06-07"
source: "https://aadhar.sh/garage/chunks"
---

> Site index: https://aadhar.sh/llms.txt
> Section index: https://aadhar.sh/garage/llms.txt
> This is the Markdown twin of a page on aadhar.sh. The HTML at the source
> URL below is the original, and is hand-written and unminified on purpose.

# Chunking & content-addressing in review

How systems store and sync big files by their *content*, and why a one-byte edit re-sends the whole file under naïve chunking, but almost nothing under [content-defined chunking][content-defined-chunking].

---

This site already runs on [content-addressing][content-addressing]: every photo is served at a content-stable URL with a `?v=N` cache-bust, the [service worker][service-worker] treats `/images/*` as immutable (cache-first), and the originals live in [R2][r2]. That's the *idea*. The systems-grade version (chunk a file by content, address each chunk by its [hash][cryptographic-hash-function], ship only the chunks that changed, and verify a slice without the whole) is what [mkit](https://github.com/officialunofficial/mkit) does. Everything below re-creates those primitives, self-contained and dependency-free, right in your browser.

## 1 · The boundary problem

Type in the box, then hit **Snapshot as v1** to freeze the current chunks. Now edit, especially **Insert a line at the top**. Watch the two strategies diverge: under **fixed-size** chunks every boundary shifts and almost every chunk's hash changes (so you'd re-upload the whole file); under **[FastCDC][fastcdc]** the boundaries ride the content, so only the chunks near your edit change.

### Fixed-size chunks every 40 bytes: the naïve approach

### FastCDC content-defined: min 16 · avg 40 · max 120 bytes

unhashed yet / no snapshot

shared with v1 (free to skip)

changed (must re-send)

Snapshot, then insert at the top: that's the whole argument for content-defined chunking in one gesture.

## 2 · FastCDC, briefly

Here is the whole idea, with every operational concern taken back out:

```
function cut(bytes, avg) {
  var mask = avg - 1, hash = 0, cuts = [];
  for (var i = 0; i < bytes.length; i++) {
    hash = (hash << 1) + GEAR[bytes[i]];
    if ((hash & mask) === 0) cuts.push(i + 1);
  }
  return cuts;
}
```

A rolling *[gear hash][gear-hash]* slides over the bytes, and wherever its low bits come up zero, that is a boundary. Because the cut points follow the *data*, inserting or deleting bytes only disturbs the boundaries you touched. The rest line up exactly as before.

Everything the shipping version adds is a separate concern wearing this function's clothes. `min`/`max` clamp pathological sizes, which is a memory budget, separate from the question of where a chunk belongs. "Normalized" chunking swaps a tighter mask in before the average and a looser one after, so sizes cluster near the target, which is a knob on the distribution. The `>>> 0` after every shift is JavaScript keeping 32-bit arithmetic honest. Peel those three off and the 60-line function is the 8 lines above, which is the version worth carrying in your head. The real one runs this page's demos: View Source and find `fastcdc`.

## 3 · Big files

Content-addressing earns its keep on large blobs. Drop any file (it never leaves your machine) and your browser FastCDC-chunks it locally (avg 8 KB), then [SHA-256][sha-256]'s each chunk into a content ID. The blocks run proportional to size so you can see the variable-length cuts; the demo counts out identical chunks (a `ChunkedBlob`'s [dedup][data-deduplication] win).

Drop a file here, or click to choose. Stays local.

## 4 · Delta wire format & verified slices

Once both sides address chunks by hash, sync is a set difference: the sender lists its chunk IDs, the receiver names the few it lacks, and only those bytes cross the wire: the **delta wire format**. The last piece is trust: **[Bao][bao] verified slices** wrap a [BLAKE3][blake3] [tree][merkle-tree] so you can stream-verify *any* byte range against the root hash without fetching the whole file. `crypto.subtle` has no BLAKE3, so the demos above use SHA-256 for the content IDs; the shape of the argument is identical, and mkit uses BLAKE3 + Bao for the real thing.

On the site: this is the same instinct as the `?v=N` immutable image URLs and the cache-first service worker: identity by content. If the photo originals ever needed incremental sync to R2, content-defined chunking is how you'd ship only what changed.

Inspired by [mkit's streaming demo](https://mkit.makechain.net/streaming) and [MakeChain](https://docs.makechain.net/): a protocol for cryptographically-signed, permissionless version control.

← back to the [garage](https://aadhar.sh/garage) · [aadhar.sh](https://aadhar.sh/)

FastCDC + content-addressing re-created in pure JS + [Web Crypto][web-crypto] · concepts credit officialunofficial/mkit

[content-defined-chunking]: https://en.wikipedia.org/wiki/Rolling_hash#Content-based_slicing_using_a_rolling_hash "Splitting a file wherever its own bytes match a pattern, so an insertion only moves the boundaries near it and the other chunks stay identical."
[content-addressing]: https://en.wikipedia.org/wiki/Content-addressable_storage "Naming each piece of data by a hash of its bytes, so the same content always gets the same name and any change gets a new one."
[service-worker]: https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API "A script a site installs in the browser that sits between its pages and the network and can answer requests from a local cache."
[r2]: https://developers.cloudflare.com/r2/ "Cloudflare's object storage for files. It speaks Amazon S3's API and charges nothing for downloading data back out."
[cryptographic-hash-function]: https://en.wikipedia.org/wiki/Cryptographic_hash_function "A short fingerprint computed from data. The same data always gives the same hash, and any change, even one byte, gives a different one."
[fastcdc]: https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia "A fast content-defined chunking algorithm from a 2016 USENIX paper. It uses a cheap gear hash and tuned masks to pick cut points."
[gear-hash]: https://en.wikipedia.org/wiki/Rolling_hash#Gear_hashing "A rolling hash that updates with one shift and one table lookup per byte, cheap enough to run over every byte of a file."
[sha-256]: https://en.wikipedia.org/wiki/SHA-2 "A hash function that turns any input into a 256-bit fingerprint. Change one byte of the input and the fingerprint changes completely."
[data-deduplication]: https://en.wikipedia.org/wiki/Data_deduplication "Storing identical pieces of data once and pointing to that one copy wherever else they appear."
[bao]: https://github.com/oconnor663/bao "A format and tool built on BLAKE3 that lets you verify any slice of a file against its root hash while streaming it."
[blake3]: https://github.com/BLAKE3-team/BLAKE3 "A fast cryptographic hash function. It hashes a file as a tree of chunks, so work runs in parallel and parts can be verified alone."
[merkle-tree]: https://en.wikipedia.org/wiki/Merkle_tree "A tree of hashes where each parent hashes its children, so one root hash vouches for every piece below it."
[web-crypto]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API "The browser's built-in cryptography toolkit for scripts, offering hashing, encryption and signatures."

Source: https://aadhar.sh/garage/chunks