Why training AI on AI data makes it worse

The problem with feeding AI its own output

The internet is filling up with AI-generated text, and that text is being scraped to train the next round of AI models. This isn't hypothetical. Researchers at Oxford and Imperial College published a 2024 paper in Nature showing what happens when a generative model trains on data produced by another generative model. It gets worse. Fast.

They call it model collapse. Some people call it "Habsburg AI," after the royal family famous for inbreeding. Others call it "AI cannibalism." When you feed a system its own output, errors compound. Variance shrinks. The model starts producing a narrower band of outputs until it's basically repeating itself.

How model collapse works

A language model learns the distribution of its training data. It doesn't memorize every word; it learns probabilities. "The cat sat on the" is more likely than "The cat sat on the refrigerator" which is more likely than "The cat sat on the quantum processor."

When you train a new model on data that came from an earlier model, you're not sampling from the original distribution. You're sampling from the model's approximation of it. That approximation loses information. Rare but real patterns get flattened. The tails of the distribution, the weird and interesting stuff, get trimmed away because they show up less often in the synthetic data.

Do this for one generation, you lose some texture. Do it for several, the model converges on a small set of high-probability outputs. The Oxford team demonstrated this empirically: after multiple rounds of training on synthetic data, their models produced nearly identical outputs regardless of the input prompt. Diversity collapsed. Quality tanked.

Why this matters right now

Two reasons. First, the data supply isn't infinite. Epoch AI published research in 2024 estimating that high-quality human text for training will be exhausted between 2025 and 2027. After that, model builders face a choice: stop scaling, or start using synthetic data.

Some companies already are. Microsoft's Phi models use it heavily. Nvidia's Nemotron. Google's Gemma fine-tuning pipeline. The pitch is "data synthesis" or "self-play," and it can work for narrow tasks. But the Nature paper shows that without careful curation and mixing with fresh human data, you hit model collapse.

Second, the web is getting polluted. A 2024 study from Amazon Web Services and University College London found that synthetic content in Common Crawl had grown significantly year over year. Train a web scrape today and you're already ingesting AI output whether you know it or not. The "clean" data problem gets harder by the month.

What helps

Accumulating data rather than replacing it: keep the original human data alongside synthetic samples and you slow the collapse. Careful filtering and scoring of synthetic outputs can preserve more variance. Reinforcement learning from human feedback (RLHF) can patch some obvious degenerations.

But none of these are free. Curating synthetic data is expensive. Keeping humans in the loop is expensive. The fundamental constraint remains: you can't create information from nothing. Every round of generation loses something. The math doesn't bend.

Watch the video

For a quick visual breakdown of how model collapse works and why it's hard to avoid, check out our short explainer:

The takeaway: synthetic data is a useful tool, not a replacement for real data. The models that will keep improving are the ones that stay connected to human input. Everyone else is just copying copies.