The Distillation Hypocrisy: Should Copying AI Models Be Legal?

What Thompson is proposing

In a July 20 Stratechery essay, Ben Thompson laid out a specific two-part law: make it explicit that collecting data to train a model is fair use, and bar terms-of-service clauses that forbid distillation — training a smaller model on a larger one's outputs — at least for U.S. companies. Simon Willison flagged the proposal as a rare attempt to resolve what he calls the hypocrisy of labs that scraped the open internet to train their own models now writing contracts forbidding anyone from doing the equivalent back to them.

Thompson's underlying argument is pragmatic: stopping distillation is nearly impossible, since it's functionally just querying an API and training on the responses. Terms-of-service bans stop nobody who actually intends to break them — they mostly bind the honest. His fix: stop pretending contract law can hold the line, and instead set policy explicitly — data collection is fair use, distillation is legal among U.S. companies, and labs get legal clarity in return.

Why this is on the table now

The essay lands right after China's two biggest open-weight releases yet. Moonshot AI shipped Kimi K3, a 2.8-trillion-parameter mixture-of-experts model (896 experts, ~16 active per token) with a 1M-token context window — Moonshot's own benchmarks place it behind Fable 5 and GPT-5.6 Sol overall but ahead of Opus 4.8 and GPT-5.5 on coding and agentic tasks, with full weights due July 27. Days later, Alibaba previewed Qwen3.8-Max, a 2.4-trillion-parameter model the company says trails only Fable 5 in its own testing, with open weights promised "soon." Both landed around the 2026 World AI Conference in Shanghai, where Xi Jinping used his keynote to explicitly back open source and international AI collaboration.

Thompson's read: Chinese labs have leaned hard on distillation over the past year, using frontier Western models as reinforcement-learning teachers to close the performance gap at a fraction of the training cost. New enforcement mechanisms are making large-scale, contract-violating distillation harder — but they won't stop a state-backed actor, and they don't help the U.S. open-model ecosystem compete on the same terms Chinese labs already operate under.

Why builders should care

For anyone shipping AI-assisted developer tools, this isn't an abstract copyright fight. Distillation is how cheaper, faster, open-weight models get made — the ones that end up self-hosted in coding agents, fine-tuned for niche tasks, or run behind a firewall for compliance reasons. If U.S. policy tilts toward Thompson's proposal, expect more legitimate domestic open-weight models built by distilling frontier output, narrowing rather than widening the gap with Kimi K3 and Qwen3.8-Max. If it doesn't, the current asymmetry — Chinese labs distilling freely, U.S. labs contractually forbidding it — persists by default.