What Is a Tokenizer? How Text Gets Split into Tokens

The term tokenizer appears everywhere as soon as you start working with language models. It affects cost, quality, speed, and how much text a model can actually handle. And yet it is often one of the least explained layers in the entire LLM stack.

The short version is simple: a tokenizer is the system that splits text into smaller pieces, tokens, and translates them into numbers the model can work with. It sounds technical, but the core idea is actually quite intuitive.

Here we go deeper into tokenization itself, why it is needed, and why models do not work directly with words or raw binary data.

What Is a Token?

A token is a piece of text that the model uses as its basic unit. Sometimes it happens to be a full word. Sometimes it is only the beginning of a word, the end of a word, a space, or punctuation.

What matters is not whether a token looks natural to us. What matters is that tokenization gives the model a practical way to represent text as a sequence of integers.

Try any text and see how it gets split up. The exact split depends on which tokenizer the model uses.

Why Use Tokens Instead of Words?

The most intuitive thing might have been to let the model work directly with words. But words are a poor technical compromise.

If every word in a language got its own ID, several problems appear immediately:

• the vocabulary explodes with inflections, compounds, names, spelling mistakes, and domain language
• new or rare words become hard to handle if they are not already in the vocabulary
• Swedish compounds, code, product names, and mixed-language text make a purely word-based system fragile

With tokens, the model can instead reuse smaller building blocks. An unusual word does not need to be unknown just because the full word has never been seen before. It can be split into parts that already exist in the vocabulary.

Why Not Let the Model Work Directly with Binary Data?

That is a reasonable question. Computers ultimately work with binary numbers, zeros and ones. So why not just feed the model raw binary data from the start?

The answer is that it becomes very inefficient.

Text is stored in computers as bits and bytes, often through encodings such as UTF-8. But if a language model worked directly at the bit level, every small piece of text would become very long. The same sentence would require many more steps, and patterns that are obvious in language would be harder for the model to detect.

One byte contains 8 bits. With 8 bits you can represent 256 different values, from 0 to 255. For simple characters like A, I, and !, one byte each is enough. In UTF-8 they therefore get the byte values 65, 73, and 33. Characters such as å, ä, and ö require more bytes.

That is why tokenization exists. It builds a more meaningful discrete representation on top of the raw data. The model still receives numbers, but at a level where language patterns are far more learnable.

Once the tokenizer vocabulary has been built, each token gets an ID. At a basic level, that is just the index of the token in the vocabulary. That is why the same text can get different token IDs in different models.

How Do You Get from Text to Token IDs?

This happens in several steps:

1. Your text is read as bytes according to a text encoding.
2. The tokenizer compares the text against its vocabulary, meaning a list of common text pieces.
3. The text is split into the pieces that best fit that vocabulary.
4. Each piece is replaced with an integer, a token ID.

So the model does not read letters or words directly. By the time the text reaches the model, it has already been turned into a sequence of numbers.

After that comes the next step in the LLM pipeline: each token ID is looked up in an embedding table and becomes a vector. That is where the model's actual mathematics begins.

How Is a Tokenizer Built?

A modern tokenizer is usually not written manually word by word. It is learned from large text corpora.

The common core idea is:

1. Start from small building blocks, often bytes or very small text units.
2. Go through huge corpora and measure which sequences appear often.
3. Merge common sequences into larger units.
4. Repeat until you have a practical vocabulary, often tens or hundreds of thousands of tokens.

That is why common words often become a single token, while unusual words can be split into several pieces. It is also why a space is sometimes included at the beginning of a token. The tokenizer has learned that certain patterns frequently appear together.

Many modern tokenizers build on ideas like byte pair encoding, or similar statistical methods. The exact algorithm varies, but the goal is the same: find a balance between a reasonably sized vocabulary and reasonably short sequences.

The point is not that someone manually decides that strings like token or izer should become tokens. They emerge because they appear often enough in the training data to become useful building blocks.

How Are Tokens Used in an LLM?

A language model does not receive text directly. The flow is basically this:

1. Text is written by the user.
2. The tokenizer converts the text into token IDs.
3. The model processes the sequence of token IDs.
4. The model predicts the next token ID.
5. The token ID is decoded back into text.

That is why LLMs are described as systems that predict the next token, not the next word. Token is the actual unit the model works with.

Why Does Tokenization Affect Cost and Context Window?

This is the practical part many people miss.

Language models are normally priced per input token and per output token. At the same time, model capacity is almost always stated as a context window in tokens. That means tokenization directly controls two central questions. If you want to go deeper into what actually fills the context, how reasoning tokens are counted, and why context engineering has become so important, see our deeper guide to the context window.

• how expensive a request becomes
• how much information the model can fit at the same time

That is also why the same amount of text does not always cost the same. A compact sentence may turn into relatively few tokens. Code, tables, JSON, mixed-language text, or long Swedish compounds can turn into many more.

Some practical consequences:

• a text with few words can still be token-heavy
• two models can produce different token counts for exactly the same text, because they use different tokenizers
• when the context window fills up, it is tokens that count, not words or characters

If you build systems with RAG, long instructions, or lots of conversation history, this quickly becomes decisive. At that point tokenization is not just theory, but actual system design.

When Should You Care a Lot About the Tokenizer?

If you only test a chatbot once in a while, you rarely need to think much about tokenization. But as soon as you build something more serious, it becomes important:

• when you want to understand or optimize cost
• when you operate close to the model's context limit
• when you send in long documents, code, or tables
• when you compare different models and see them behave differently despite the same input
• when you build RAG or agent flows where a lot of text passes through the model

Summary

A tokenizer is the translation layer between human text and the model's mathematics. It splits text into tokens and maps them to integers the model can work with.

That is why language models do not work directly with words, and not with raw binary data either. Words are too rigid. Binary data is too low-level and too inefficient. Tokens are the compromise that makes modern LLMs practical.

And when someone says a model has a context window of 128,000, they always mean tokens.