1,749 language models, and vocabulary size predicts nothing

The plain-language companion to "The Receiver-Limited Floor" (Zenodo, 2026). No math required. The full works are one click away at the bottom.

If you've been anywhere near the AI conversation lately, you've heard the one commandment everybody chants: bigger. More parameters, more data, more everything. The whole industry runs on the cheerful assumption that if a little is good, a lot is better, and a whole lot is a press release.

So we went and poked one particular version of that belief with a stick. The belief was this: give an AI a bigger vocabulary — more word-chunks to work with — and it'll get smarter. Reasonable-sounding, isn't it? A person with a bigger vocabulary usually is sharper. We tested it on 1,749 different AIs. The belief did not survive.

The Biggest Menu Survey Ever Run

Seventeen hundred and forty-nine language models — as far as we know, the largest roundup of its kind anybody's ever bothered to do. Little ones, giant ones, translators, chatbots. Vocabularies ranging from a few thousand chunks to a quarter of a million. And for each one we measured a simple thing: how tightly can it compress a passage of English? (Tighter compression means the machine genuinely gets the language, the way a person who understands a joke can retell it in fewer words.)

Then the one question that mattered: does a bigger vocabulary predict a smarter machine?

Behold, a Flat Line

No. Not a little bit no — emphatically no. The connection between vocabulary size and smarts came back at essentially zero. In the cold statistical terms: vocabulary size explains less than one percent of the difference between a good model and a bad one. Draw it on a graph and you get a line as flat as Kansas. A model with 32,000 word-chunks does about as well as one with 128,000. The extra hundred thousand words bought precisely nothing.

Here's the homey version. Imagine you want to know whether the length of a restaurant's menu predicts how good the food is. You dutifully visit 1,749 restaurants. The verdict comes back: menu size explains 0.7% of how tasty the meal is. The menu, it turns out, is a decoy. What you actually wanted to ask about was the chef.

Why? Because the Listener Sets the Limit

Here is the heart of it, and it's the kind of idea that rearranges your furniture a little once it's in your head. A vocabulary is just a bag of labels. Making the bag bigger gives you more ways to say a thing — but it doesn't give you more things worth saying, and it doesn't help the listener on the other end hear you any better. The bottleneck was never the labels. It's the listener.

Picture yourself at a loud party, straining to catch what a friend is saying over the din. Would a bigger dictionary help? Of course not — you're not stuck for words, you're drowning in noise. The racket in the room decides how much you can actually make out per sentence, and no amount of extra vocabulary changes the racket. That racket has a name in the math — it's the fundamental noise of the channel — and it, not the size of your word-bag, sets the ceiling on how much real meaning gets through per step. Claude Shannon worked out the equations for exactly this back in the 1940s, and the AIs are simply obeying them, whether they know it or not.

The Ribosome Nods Along

Now the part that gave us goosebumps. Remember that little machine from our first story, the ribosome, reading life's four-letter book? It faces the very same wall.

The ribosome has a tiny "vocabulary" — its sixty-four codons, sorting out twenty-odd amino acids — and it squeezes out about 4.4 bits of real information per step. Could it do better with a fatter vocabulary? No. A cleverer code? No. The floor is the floor. And here's the rhyme: today's AIs, with vocabularies a thousand times bigger than the ribosome's, built from silicon instead of protein, with nothing whatsoever in common historically — also come to rest in that same cramped little band of a few bits per step.

We should be honest about exactly what kind of rhyme this is, because it's easy to oversell and we'd rather not. The AI's few-bits number is measured per chunk-of-text; the ribosome's is per chemical decision. They're cousins, not twins — the same neighborhood, not the identical house number. But the striking thing stands: a thousandfold spread in vocabulary, and everybody still ends up on the same quiet street. The vocabulary didn't matter. The listener did. And both listeners — the one made of protein and the one made of math — bumped into a ceiling in roughly the same place.

What This Means If You Build the Things

If you happen to make AI for a living, the moral arrives with a bill attached: quit fiddling with the vocabulary and go fix the listener. Every hour poured into clever tokenizer tricks is an hour not spent on the thing that actually moves the needle — making each step of thinking squeeze more meaning out of the world. The data is blunt: at any sensible size, vocabulary is a solved problem. The hard, unsolved, interesting problem is the listener's efficiency — and there may be a hard wall there too.

Same story for the hardware crowd and the folks paying the electric bill: it's not the granularity of the chopping that costs you, it's the size of the machine doing the thinking. Optimize joules-per-good-decision, not chunks-per-second.

The Bigger Lesson

This was the second time we watched wildly different systems — a protein machine and a silicon one — trudge up to the exact same wall and stop. Life's code: a handful of bits per step. A giant modern AI: a handful of bits per step. A thousandfold difference in vocabulary buying almost no difference in throughput at all.

The universe, it seems, doesn't care how big your dictionary is. It cares about the noise, and it cares about how hard it is to tell one thing from another. Which raised an irresistible question: how many other systems are quietly obeying this same wall? Nature, music, memory, machines? That headcount is Paper 3 — and the tally is bigger than you'd think.

The Receiver-Limited Floor is Paper 2 of the Windstorm series.
Zenodo: doi.org/10.5281/zenodo.19322972 · Code & data: github.com/Windstorm-Institute/receiver-limited-floor
Download the full paper (PDF)