What Are Tokens in LLMs? Understanding How AI Language Models Actually Work
Updated: September 17, 2026
I learned what a "token" was by watching an LLM ignore a simple persona and promote my $15k/year janitor to "CEO." That bug sent me down the rabbit hole of tokenization: why models don't read words the way we do, how BPE/WordPiece/SentencePiece slice text into chunks, and why "next-token prediction" explains both the magic and the lies. If you've ever hit a context limit or watched a model forget what you just told it, this is the real reason. read on »