Tokenization converts raw text into a sequence of discrete units (tokens) and maps each to an integer ID that a model can embed. It is the first step of every NLP pipeline and every LLM call, and it quietly shapes model quality, context-window usage, cost (APIs bill per token), multilingual fairness, and odd failure modes such as miscounting letters or digits. Modern systems almost all use subword tokenizers learned from data: BPE, WordPiece or Unigram.
Part of Natural Language Processing. Downstream tasks include Text Classification and Named Entity Recognition.
Libraries such as Hugging Face tokenizers structure the process as four stages. Normalization cleans text (Unicode normalization, accent stripping, lowercasing for uncased models). Pre-tokenization splits text into rough word-like chunks so merges never cross those boundaries; GPT-style tokenizers use a regex that keeps a leading space attached to the following word. The model splits each chunk into vocabulary subwords. Post-processing adds special tokens. The output includes IDs, attention masks and character offsets, which are essential for mapping predictions back to text in tasks like NER.
Word-level tokenization (split on whitespace and punctuation) is intuitive but needs a very large vocabulary and still meets out-of-vocabulary words, especially in morphologically rich languages. Character- or byte-level tokenization never has unknowns but makes sequences several times longer, which is expensive for attention whose cost grows quadratically with length. Subword tokenization is the compromise: a fixed vocabulary (commonly tens of thousands to a few hundred thousand entries) in which common words are single tokens and rare words are composed from pieces.
Byte-Pair Encoding, adapted for NMT by Sennrich et al. (2016), starts from characters and repeatedly merges the most frequent adjacent pair in the training corpus, adding the merged symbol to the vocabulary, until the target vocabulary size is reached. The result is an ordered list of merge rules. Encoding applies those merges, in order, to each pre-tokenized chunk. Byte-level BPE (GPT-2 onward, and tiktoken-based tokenizers) runs BPE over UTF-8 bytes rather than Unicode characters, so the 256 base bytes guarantee every possible string is representable with no unknown token.
| Algorithm | How it builds the vocabulary | Used by (examples) |
|---|---|---|
| BPE / byte-level BPE | Greedy frequency-based merges, bottom-up | GPT family, RoBERTa, LLaMA 3, many open LLMs |
| WordPiece | Merges chosen by likelihood gain (pair frequency normalized by part frequencies); continuation pieces marked ## | BERT, DistilBERT |
| Unigram LM | Starts large and prunes tokens that least reduce corpus likelihood; can sample alternative segmentations | T5, ALBERT, XLNet (via SentencePiece) |
SentencePiece is a library rather than an algorithm: it implements BPE and Unigram directly on raw text, treating whitespace as an ordinary symbol (shown as ▁). Because it needs no language-specific pre-tokenizer, it works well for languages without spaces such as Japanese and Chinese, and decoding is exactly reversible. Unigram's ability to sample segmentations enables subword regularization, a data-augmentation trick that improves robustness.
# pip install tokenizers transformers tiktoken
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, normalizers
corpus = [
"low lower lowest newest widest",
"the newest model tokenizes the widest range of text",
"tokenization splits rare words into subword units",
] * 200
tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.normalizer = normalizers.NFKC()
tok.pre_tokenizer = pre_tokenizers.Whitespace()
trainer = trainers.BpeTrainer(vocab_size=200, special_tokens=["[UNK]", "[PAD]"])
tok.train_from_iterator(corpus, trainer)
enc = tok.encode("the lowest newest tokenizer")
print(enc.tokens) # subword pieces
print(enc.ids)
print(enc.offsets) # (start, end) character spans
# Pretrained tokenizers
from transformers import AutoTokenizer
bert = AutoTokenizer.from_pretrained("bert-base-uncased")
print(bert.tokenize("Tokenization is unbelievably useful")) # '##' marks continuations
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Count tokens before you call an API.")
print(len(ids), [enc.decode([i]) for i in ids])
Always load the tokenizer that matches the model checkpoint. Using a different tokenizer, even one with the same vocabulary size, produces IDs that mean different things to the model.
"hello" and " hello" are different tokens, which affects prompt construction and stop sequences.-100).<|endoftext|>) should be encoded as plain text, not as control tokens.The classic toy corpus from Sennrich et al. has four words with counts: low (5), lower (2), newest (6), widest (3). Split into characters, the most frequent adjacent pairs are e s and s t (9 each, from newest and widest), then w e (8), then l o and o w (7 each).
e s → es. Pair counts are recomputed; es t now occurs 9 times.es t → est. A useful suffix is now a single symbol.l o → lo (7).lo w → low (7).The unseen word lowest is encoded by replaying these merges in order, giving low + est: two tokens and no unknown symbol, even though the word never appeared in training. Two details matter in real implementations. Ties must be broken deterministically or two runs produce different vocabularies. And at encode time the merges are applied by rank (earliest-learned first), not by re-counting frequencies, which is why the merge list is part of the tokenizer file.
Token counts are what you pay for and what fills the context window, so it is worth measuring them rather than guessing. OpenAI's published rule of thumb for English with its tokenizers is roughly four characters, or about three quarters of a word, per token. That ratio does not transfer to other languages, code or numbers.
1234 may be one token, 1235 two). Some model families split digits individually or into fixed-size groups so that arithmetic patterns are learnable.To compare tokenizers, take a representative sample of your own traffic, count tokens per character for each candidate, and look at the distribution, not just the mean: long-tail inputs such as tables, logs or base64 blobs are often where the cost surprises are.
Beyond ordinary vocabulary entries, every model reserves special tokens that carry structure: BERT's [CLS], [SEP], [PAD] and [MASK]; sequence markers such as <s> and </s>; and, in chat models, role and turn delimiters. A chat template (in Hugging Face, tokenizer.apply_chat_template) renders a list of messages into exactly the token layout the model was fine-tuned on. Formatting the prompt by hand with slightly different delimiters is a common, silent cause of degraded output.
Special tokens are also a security boundary. If user-supplied text containing the literal string of a control token is encoded as that control token, a user can forge turn boundaries. Libraries guard against this: tiktoken, for instance, raises an error by default when input contains special-token text, and Hugging Face tokenizers can be told not to split special tokens out of untrusted text. Treat that setting as part of your prompt-injection defenses.
Finally, when you add domain tokens to an existing vocabulary, the new embedding rows start untrained; resize the model's embedding matrix, initialize the new rows sensibly (for example the mean of the subword embeddings they replace) and fine-tune, or the model will treat them as noise.
tokenizers documentation and the tiktoken README.