---
title: "What Are Tokens in LLMs? Understanding How AI Language Models Actually Work"
description: "Learn what tokens really are in LLMs and why AI models hallucinate. Discover how tokenization works, why context windows matter, and how understanding tokens helps you build better AI systems that don't confuse janitors with CEOs."
image: "https://www.sethserver.com/static/images/what-are-tokens-in-nlp-and-llms-social.jpg"
---

# What Are Tokens in LLMs? Understanding How AI Language Models Actually Work

I learned what a "token" really was the hard way: by watching an LLM confidently lie to my face.

This was a few years ago. I was building an AI survey response system for a client. The pitch sounded clean: generate personas with fixed attributes, then have them answer survey questions *as that persona*. So I'd set something like:

- Job title: **janitor**
- Salary: **$15,000/year**

Then I'd ask: "What's your job title?"

And the model would answer: "CEO."

"What's your salary?"

"$35,000/year."

Not wrong in a clever way. Wrong in a "did you read any of that?" way.

That was my first real lesson in tokens. The model isn't *being* a janitor. It's predicting the next token that looks likely given the tokens it has already seen. "Job title" sits near "CEO" in a lot of training data. Same for "salary" and numbers that look more normal than $15k. The model is not checking a little internal spreadsheet of persona facts. It's doing math on token probabilities.

## Tokens: the model's alphabet (but not your alphabet)

A **token** is the unit of text the model actually reads and writes. Not always words. Sometimes whole words like `machine`. Sometimes chunks like `learn` and `##ing`. Sometimes punctuation. Sometimes weird whitespace artifacts (yes, really). Sometimes special control symbols that mean "beginning of text" or whatever.

A common example looks like this:

```text
"I love machine learning!"
["I", "love", "machine", "learn", "##ing", "!"]
```

That `##ing` is a clue: lots of tokenizers use **subword tokens** so they can reuse common fragments instead of storing every possible word.

## Why tokenization exists (and why it bites you)

Tokenization is a compromise between two bad extremes.

If you tokenize by **word**, it feels natural, but your vocabulary explodes. Also, a single weird word like "unicornification" turns into an out-of-vocabulary mess.

If you tokenize by **character**, your vocabulary is tiny, but sequences get long fast. The model has to do more work to learn meaning across many small steps.

Subwords are the middle path. They keep the vocabulary manageable while still letting the model represent weird words by stitching pieces together.

Here's why this actually matters in practice: the model needs a consistent input format. A fixed vocabulary keeps training and inference computationally sane. The model learns meaning from token patterns, not raw characters. When it hits a word it doesn't know-a name, a typo, new slang-it breaks it into chunks it does know. And subwords sometimes help across languages, though that's messier than it sounds.

## The tokenizers everyone uses (and why)

You'll run into a few names over and over.

### Byte-Pair Encoding (BPE)

**Byte-Pair Encoding (BPE)** starts with characters, then keeps merging the most common pairs. You end up with a vocabulary of frequent chunks. I like it for handling rare words without the vocab exploding to infinity.

### WordPiece

**WordPiece** is in the same family, but it chooses merges based on what helps prediction, not just raw frequency. BERT uses this. You'll also see the `##` prefix style a lot in WordPiece tokenizers.

### SentencePiece

**SentencePiece** is the practical one. It doesn't assume spaces mean word boundaries, which is great for languages where that's not true. It treats everything as a stream of Unicode characters. I've used it a few times and the docs are actually readable, which is saying something.

## How LLMs use tokens: probability all the way down

Once you've got tokens, each one becomes a vector-basically a point in high-dimensional space. Similar tokens cluster together.

Then Transformers add positional encoding (so the model knows what order things came in) and attention (so it can figure out which earlier tokens matter for what it's doing right now).

Generation is simple in theory: compute probabilities for every possible next token, pick one (greedy, sampled, temperature, top-p, whatever), then repeat until you hit a stop token or run out of space.

That's the whole trick. A next-token machine with a very expensive education.

It's also why my "janitor" became a "CEO." The model saw tokens that often co-occur with job-and-salary questions and took the path that usually works in the wild.

## Token limits: the quiet constraint that shapes everything

Early on I was building against ChatGPT 3.5 with a small context window. A few thousand tokens and you were done. You'd watch the model forget things you *just told it* because those tokens got pushed out of context.

This is where theory stops being cute. You hit the limit in production and suddenly you care a lot about chunking, summaries, and retrieval. Nothing motivates learning like a bug report.

## The useful takeaway (even if it's a little deflating)

Understanding tokens explains a lot:

- Why LLMs feel smart: they've seen tons of patterns.
- Why they hallucinate: nothing anchors them unless you force it.
- Why they're not really "creative": they remix probabilities.
- And honestly, why humans aren't that original either. Most of what we say is [autocomplete with better context](https://www.sethserver.com/ai/ai-writes-a-little-too-well-but-somehow-not-well-enough-the-uncanny-valley-of-llms.html).

Once I understood tokens, I stopped expecting the model to "know" anything. Now I design systems that force it to behave: structured prompts, tool calls, retrieval when it needs facts, validation so it can't just make stuff up, and constraints that keep it from wandering into CEO-land.

I still get LLMs that insist my $15k/year janitor is a CEO. But now I know it's not the model being stupid. It's just doing what it was trained to do: predict the tokens that usually come next. Turns out "job title" and "CEO" sit real close in the training data.

-Sethers


```json
{"@context": "https://schema.org", "@type": "BlogPosting", "headline": "What Are Tokens in LLMs? Understanding How AI Language Models Actually Work", "datePublished": "2024-10-20T04:37:21", "dateModified": "2026-09-17T19:44:24", "image": {"@type": "ImageObject", "url": "https://www.sethserver.com/static/images/what-are-tokens-in-nlp-and-llms-social.jpg", "width": 400, "height": 400}, "url": "https://www.sethserver.com/ai/what-are-tokens-in-nlp-and-llms.html", "author": {"@type": "Person", "name": "Seth Black", "url": "https://www.linkedin.com/in/seth-black-tx"}, "publisher": {"@type": "Organization", "name": "Sethserver", "logo": {"@type": "ImageObject", "url": "https://www.sethserver.com/static/images/PaperAirplane512.png"}}, "description": "Learn what tokens really are in LLMs and why AI models hallucinate. Discover how tokenization works, why context windows matter, and how understanding tokens helps you build better AI systems that don't confuse janitors with CEOs.", "articleSection": "Technology", "articleBody": "In the world of Natural Language Processing (NLP) and Large Language Models (LLMs), tokens are the unsung heroes that make it all possible. They're the linguistic Legos that allow machines to understand and generate human language with surprising accuracy. But what exactly are tokens, and why are they so important? Let's dive in and demystify these fundamental building blocks of modern AI language systems.  The Basics: What is a Token?  At its core, a token is a unit of text that an NLP model or LLM processes as a single entity. But here's where it gets interesting: tokens aren't always what you might expect. They're not necessarily words, and they're definitely not just individual characters. Tokens can be words, parts of words, punctuation marks, or even special symbols that represent specific concepts or functions within the model.  For example, in the sentence \"I love machine learning!\", each word might be a separate token: [\"I\", \"love\", \"machine\", \"learning\", \"!\"]. But in more advanced tokenization schemes, it could look something like this: [\"I\", \"love\", \"machine\", \"learn\", \"##ing\", \"!\"], where \"##ing\" is a subword token representing a common suffix.  The key thing to understand is that tokens are flexible units designed to balance the need for granularity (breaking text down into small, manageable pieces) with the need for meaningful semantic units (preserving the meaning and structure of language).  Why Tokens Matter  Tokens serve several crucial functions in NLP and LLMs:  1. Input Processing: They provide a standardized way to feed text into a model, breaking down the infinite variety of human language into a finite set of discrete units.  2. Vocabulary Management: By using tokens, models can work with a fixed vocabulary size, which is essential for computational efficiency and memory management.  3. Semantic Understanding: Well-designed tokenization schemes can capture semantic information, helping models understand the meaning of words in context.  4. Out-of-Vocabulary Handling: Tokens allow models to handle words they've never seen before by breaking them down into familiar subword units.  5. Multilingual Support: Some tokenization methods enable models to work across multiple languages without requiring separate training for each one.  Tokenization Methods: From Simple to Sophisticated  Now that we understand what tokens are and why they're important, let's look at some common tokenization methods, from the simplest to the more advanced techniques used in state-of-the-art LLMs.  1. Word Tokenization  This is the most straightforward approach: split the text on whitespace and punctuation. It's simple and intuitive but has some significant drawbacks. It struggles with compound words, doesn't handle out-of-vocabulary words well, and can result in very large vocabularies, especially for morphologically rich languages.  2. Character Tokenization  At the other extreme, we could treat each character as a token. This drastically reduces the vocabulary size and eliminates the out-of-vocabulary problem, but it loses all word-level semantic information and results in very long sequences of tokens for any given text.  3. Subword Tokenization  This is where things get interesting. Subword tokenization methods strike a balance between word and character tokenization. They break words into meaningful subword units, allowing models to understand parts of words and recombine them in novel ways. Two popular subword tokenization methods are:     a) Byte-Pair Encoding (BPE): This method starts with a character-level tokenization and iteratively merges the most frequent pairs of tokens to form new tokens. It's great for handling rare words and compound words.     b) WordPiece: Similar to BPE, but it chooses merges based on the likelihood of improving the language model's probability, rather than just frequency. It's used in models like BERT.  4. SentencePiece  This is an unsupervised text tokenizer and detokenizer, primarily designed for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. It treats the input as a sequence of Unicode characters, which allows it to be language-independent.  Tokens in Action: How LLMs Use Them  Now that we understand what tokens are and how they're created, let's look at how Large Language Models actually use them.  1. Token Embeddings  When a piece of text is fed into an LLM, each token is first converted into a token embedding - a dense vector representation of the token in a high-dimensional space. These embeddings capture semantic relationships between tokens, allowing the model to understand that \"dog\" and \"puppy\" are more closely related than \"dog\" and \"skyscraper\", for example.  2. Positional Encoding  In many modern LLM architectures, like Transformers, the order of tokens is crucial. But these models process all tokens in parallel, so they need a way to know each token's position in the sequence. This is where positional encoding comes in, adding position information to each token's embedding.  3. Attention Mechanisms  LLMs use attention mechanisms to weigh the importance of different tokens when processing or generating text. This allows the model to focus on relevant parts of the input when producing each part of the output.  4. Token Generation  When generating text, LLMs produce one token at a time. At each step, the model predicts the probability distribution over its entire vocabulary, and the next token is chosen based on this distribution (often with some randomness added for creativity).  Tokens in Context: Real-World Examples  Let's look at how tokens might be used in some real-world NLP tasks:  1. Machine Translation  When translating from English to French, a model might break down the English sentence into tokens, process them through its layers, and then generate French tokens one by one. The subword nature of many tokenization schemes helps handle words that might not have direct translations.  2. Sentiment Analysis  In sentiment analysis, certain tokens might be strongly associated with positive or negative sentiment. The model learns these associations during training, allowing it to make accurate predictions even for sentences it hasn't seen before.  3. Text Completion  When you're typing in a search engine or using a smart compose feature, the model is predicting the next token based on the tokens you've already typed. This is why these systems can often complete your thought in a surprisingly accurate way.  4. Question Answering  In a question-answering system, the model processes tokens from both the question and a given context passage. It then generates tokens for the answer, drawing information from the relevant parts of the context.  Challenges and Limitations of Current Tokenization Approaches  While tokens have revolutionized NLP and enabled the creation of incredibly powerful language models, they're not without their challenges:  1. Out-of-Vocabulary Words  Even with subword tokenization, models can sometimes encounter truly novel words or names that they struggle to tokenize effectively.  2. Context-Dependent Tokenization  The meaning of a word can change based on context, but most tokenization methods don't take this into account. For example, \"bank\" in \"river bank\" versus \"bank account\" might ideally be tokenized differently.  3. Cross-Lingual Challenges  While some tokenization methods work well across multiple languages, creating truly language-agnostic tokens remains a challenge, especially for languages with very different writing systems.  4. Token Limits  Most LLMs have a maximum number of tokens they can process at once (often around 512 or 1024). This can be limiting for tasks that require understanding long documents or conversations.  5. Lossy Compression  Tokenization is inherently a form of lossy compression. Some nuance or information from the original text may be lost in the process of breaking it down into tokens.  The Future of Tokens  As NLP and LLM technology continues to advance, we're likely to see further innovations in tokenization:  1. Dynamic Tokenization  Future models might adjust their tokenization on the fly based on context, allowing for more nuanced understanding of language.  2. Multimodal Tokens  As AI moves towards processing multiple types of data (text, images, audio), we might see the development of tokenization schemes that can represent information across different modalities.  3. Neurosymbolic Approaches  Some researchers are exploring ways to combine neural network-based approaches with symbolic AI, which could lead to new ways of representing and processing language that go beyond current token-based methods.  Conclusion  Tokens are the unassuming workhorses of modern NLP and LLMs. They bridge the gap between human language and machine understanding, enabling AI systems to process and generate text with impressive capabilities. By breaking down the complexity of language into manageable, meaningful units, tokens allow models to capture the nuances of human communication in a way that's computationally feasible.  As we've seen, tokens are more than just words or characters - they're flexible, context-aware building blocks that form the foundation of some of the most advanced AI systems in the world. From simple word splitting to sophisticated subword tokenization methods, the evolution of tokenization techniques has played a crucial role in the rapid advancements we've seen in NLP over the past few years.  Understanding tokens gives us insight into how LLMs \"think\" and process language. It helps explain both their remarkable capabilities and their occasional quirks or limitations. As developers, researchers, or simply curious individuals, grasping the concept of tokens allows us to better understand, utilize, and improve these powerful AI tools.  As we look to the future, tokens will undoubtedly continue to evolve, enabling even more sophisticated language understanding and generation. Who knows? The next breakthrough in AI might just come from a novel approach to breaking down and representing human language.  So the next time you're amazed by an AI's ability to understand or generate human-like text, spare a thought for the humble token - the linguistic Lego that makes it all possible.  -Sethers"}
{"@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "sethserver", "item": "https://www.sethserver.com/"}, {"@type": "ListItem", "position": 2, "name": "AI", "item": "https://www.sethserver.com/ai/"}, {"@type": "ListItem", "position": 3, "name": "What Are Tokens in LLMs? Understanding How AI Language Models Actually Work", "item": "https://www.sethserver.com/ai/what-are-tokens-in-nlp-and-llms.html"}]}
```
