Simon Willison's SQLite Trick: Vibecoding a 250x History Compressor

The idea: compress the whole history together, not version by version

Simon Willison — the developer behind Datasette and the llm CLI — has a recurring interest in how to store revision histories in relational databases. His latest angle, written up on his blog, came to him on a dog walk: instead of diffing each revision against the last one, store the *full text* of every prior version as a big JSON array of strings, then run the whole array through zlib or zstd. The bet is that generic compression should do well here because consecutive revisions of the same document share huge amounts of repeated text.

From voice mode to a working benchmark in 38 minutes

Willison first talked the idea through using OpenAI's GPT-Live voice mode in the ChatGPT iPhone app — useful for sanity-checking a concept, but voice conversations still can't be shared as URLs. He then switched to text and gave a precise prompt to GPT-5.6 Sol Pro, asking it to build and benchmark the idea. The model took 38 minutes and returned code plus a written answer, all published in this GitHub folder.

The prototype compares two storage designs: WholeBlobHistoryStore, which compresses the entire revision array as one blob, and ChunkedHistoryStore, which splits history into rows capped at either 128 revisions or 3MB of uncompressed JSON — a hedge against a single document's history growing into one unwieldy row. Timestamps are kept out of the compressed blob, stored separately since they don't compress the same way as repeated text.

• 1,000 simulated revisions produced 20.4 MB of raw revision text
• Compressed as a single Zstandard JSON array, that came to 80.3 KB — roughly 250x smaller
• Chunking caps: 128 revisions or 3MB of uncompressed JSON per row, whichever comes first

Why this is a good template, not just a good hack

The compression result is a nice number, but the more useful thing here is the process. Willison didn't ask an assistant to "build a revision history system" in the abstract — he formed a specific, testable hypothesis (repeated strings compress well across versions), talked it into a clear shape using voice, then handed a coding model a scoped spec it could execute and benchmark unsupervised. The output wasn't taken on faith; it shipped with real numbers on real simulated data, and the code is public for anyone to rerun.

That loop — articulate the idea, write a spec tight enough that an LLM can't wander, then verify against measured output rather than the model's own claims — is the difference between vibecoding a toy and vibecoding something you'd actually trust in a codebase. It applies just as well to areas builders in fintech and trading infrastructure care about: audit trails, config history, and order or strategy version logs in SQLite-backed systems all share the same property Willison is exploiting — successive versions of the same record tend to differ by very little.

The caveats worth keeping in mind

This is a research prototype, not a shipped library — Willison's post frames it as an experiment, and the benchmark uses simulated revisions rather than a production workload. Whole-blob compression also means reading or writing touches the full history blob, which is a real tradeoff against chunking, indexing, or a dedicated diff format depending on how often old revisions actually get read.