sethserver.com Subscribe
None

What Are Tokens in LLMs? Understanding How AI Language Models Actually Work

By Seth Black • Updated: September 17, 2026

AI · 4 min read

Like this kind of writing? Get one email a week with notes on startups, AI, and the occasional strong opinion about Python: subscribe to the newsletter.

I learned what a "token" really was the hard way: by watching an LLM confidently lie to my face.

This was a few years ago. I was building an AI survey response system for a client. The pitch sounded clean: generate personas with fixed attributes, then have them answer survey questions as that persona. So I'd set something like:

Then I'd ask: "What's your job title?"

And the model would answer: "CEO."

"What's your salary?"

"$35,000/year."

Not wrong in a clever way. Wrong in a "did you read any of that?" way.

That was my first real lesson in tokens. The model isn't being a janitor. It's predicting the next token that looks likely given the tokens it has already seen. "Job title" sits near "CEO" in a lot of training data. Same for "salary" and numbers that look more normal than $15k. The model is not checking a little internal spreadsheet of persona facts. It's doing math on token probabilities.

Tokens: the model's alphabet (but not your alphabet)

A token is the unit of text the model actually reads and writes. Not always words. Sometimes whole words like machine. Sometimes chunks like learn and ##ing. Sometimes punctuation. Sometimes weird whitespace artifacts (yes, really). Sometimes special control symbols that mean "beginning of text" or whatever.

A common example looks like this:

"I love machine learning!"
["I", "love", "machine", "learn", "##ing", "!"]

That ##ing is a clue: lots of tokenizers use subword tokens so they can reuse common fragments instead of storing every possible word.

Why tokenization exists (and why it bites you)

Tokenization is a compromise between two bad extremes.

If you tokenize by word, it feels natural, but your vocabulary explodes. Also, a single weird word like "unicornification" turns into an out-of-vocabulary mess.

If you tokenize by character, your vocabulary is tiny, but sequences get long fast. The model has to do more work to learn meaning across many small steps.

Subwords are the middle path. They keep the vocabulary manageable while still letting the model represent weird words by stitching pieces together.

Here's why this actually matters in practice: the model needs a consistent input format. A fixed vocabulary keeps training and inference computationally sane. The model learns meaning from token patterns, not raw characters. When it hits a word it doesn't know-a name, a typo, new slang-it breaks it into chunks it does know. And subwords sometimes help across languages, though that's messier than it sounds.

The tokenizers everyone uses (and why)

You'll run into a few names over and over.

Byte-Pair Encoding (BPE)

Byte-Pair Encoding (BPE) starts with characters, then keeps merging the most common pairs. You end up with a vocabulary of frequent chunks. I like it for handling rare words without the vocab exploding to infinity.

WordPiece

WordPiece is in the same family, but it chooses merges based on what helps prediction, not just raw frequency. BERT uses this. You'll also see the ## prefix style a lot in WordPiece tokenizers.

SentencePiece

SentencePiece is the practical one. It doesn't assume spaces mean word boundaries, which is great for languages where that's not true. It treats everything as a stream of Unicode characters. I've used it a few times and the docs are actually readable, which is saying something.

How LLMs use tokens: probability all the way down

Once you've got tokens, each one becomes a vector-basically a point in high-dimensional space. Similar tokens cluster together.

Then Transformers add positional encoding (so the model knows what order things came in) and attention (so it can figure out which earlier tokens matter for what it's doing right now).

Generation is simple in theory: compute probabilities for every possible next token, pick one (greedy, sampled, temperature, top-p, whatever), then repeat until you hit a stop token or run out of space.

That's the whole trick. A next-token machine with a very expensive education.

It's also why my "janitor" became a "CEO." The model saw tokens that often co-occur with job-and-salary questions and took the path that usually works in the wild.

Token limits: the quiet constraint that shapes everything

Early on I was building against ChatGPT 3.5 with a small context window. A few thousand tokens and you were done. You'd watch the model forget things you just told it because those tokens got pushed out of context.

This is where theory stops being cute. You hit the limit in production and suddenly you care a lot about chunking, summaries, and retrieval. Nothing motivates learning like a bug report.

The useful takeaway (even if it's a little deflating)

Understanding tokens explains a lot:

Once I understood tokens, I stopped expecting the model to "know" anything. Now I design systems that force it to behave: structured prompts, tool calls, retrieval when it needs facts, validation so it can't just make stuff up, and constraints that keep it from wandering into CEO-land.

I still get LLMs that insist my $15k/year janitor is a CEO. But now I know it's not the model being stupid. It's just doing what it was trained to do: predict the tokens that usually come next. Turns out "job title" and "CEO" sit real close in the training data.

-Sethers

Share this post
Newsletter

One email, once a week.

Notes on databases, systems, and the occasional strong opinion about Python. No spam, unsubscribe anytime.

Seth Black
Written by

Seth Black

Engineer and founder based in Texas. Writes about databases, AI, and running things in production. Embeds in small teams as lead engineer.

More from AI

View all →
AI

How to be a Good Farrier: What the Car Teaches Us About AI and Your Job

Nov 10, 2026
AI

Principle of Least Privilege in the AI Age

Apr 14, 2026
AI

How to Actually Secure MCP (Model Context Protocol)

Apr 14, 2026