Tokens in NLP: Text Tokenization Guide

08 of december of 2023

Disclaimer: This post has been translated to English using a machine translation model. Please, let me know if you find any mistakes.

Now that LLMs are on the rise, we keep hearing about the number of tokens each model supports, but what are tokens? They are the smallest units of representation of words.

To explain what tokens are, let's first look at a practical example using the OpenAI tokenizer, called tiktoken.

So, first we install the package:

pip install tiktoken

Once installed, we create a tokenizer using the cl100k_base model, which in the example notebook How to count tokens with tiktoken explains is used by the models gpt-4, gpt-3.5-turbo and text-embedding-ada-002

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		import tiktoken
 
encoder = tiktoken.get_encoding("cl100k_base")
	
	Copied

Now we create an example word to tokenize it

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		example_word = "breakdown"
	
	Copied

And we tokenize it

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		tokens = encoder.encode(example_word)
tokens
	
	Copied

>_ Output

			
				[9137, 2996]

The word has been split into 2 tokens, the 9137 and the 2996. Let's see which words they correspond to.

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		word1 = encoder.decode([tokens[0]])
word2 = encoder.decode([tokens[1]])
word1, word2
	
	Copied

>_ Output

			
				('break', 'down')

The OpenAI tokenizer has split the word breakdown into the words break and down. That is, it has divided the word into 2 simpler ones.

This is important, as when it is said that an LLM supports x tokens, it does not mean that it supports x words, but rather that it supports x minimal units of word representation.

If you have a text and want to see the number of tokens it has for the OpenAI tokenizer, you can check it on the Tokenizer page, which displays each token in a different color.

We have seen the tokenizer of OpenAI, but each LLM may use a different one.

As we have said, the tokens are the minimal units of representation of words, so let's see how many distinct tokens tiktoken has.

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		n_vocab = encoder.n_vocab
print(f"Vocab size: {n_vocab}")
	
	Copied

>_ Output

			
				Vocab size: 100277

Let's see how it tokenizes another type of words

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		def encode_decode(word):
    tokens = encoder.encode(word)
    decode_tokens = []
    for token in tokens:
        decode_tokens.append(encoder.decode([token]))
    return tokens, decode_tokens
	
	Copied

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		word = "dog"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "tomorrow..."
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "artificial intelligence"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "Python"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "12/25/2023"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "😊"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
	
	Copied

>_ Output

			
				Word: dog ==&gt; tokens: [18964], decode_tokens: ['dog']
Word: tomorrow... ==&gt; tokens: [38501, 7924, 1131], decode_tokens: ['tom', 'orrow', '...']
Word: artificial intelligence ==&gt; tokens: [472, 16895, 11478], decode_tokens: ['art', 'ificial', ' intelligence']
Word: Python ==&gt; tokens: [31380], decode_tokens: ['Python']
Word: 12/25/2023 ==&gt; tokens: [717, 14, 914, 14, 2366, 18], decode_tokens: ['12', '/', '25', '/', '202', '3']
Word: 😊 ==&gt; tokens: [76460, 232], decode_tokens: ['�', '�']

Finally, we will see it with words in another language

	
		
			< >
			Input
		
		
			Python
			
		
	
	
		word = "perro"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "perra"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "mañana..."
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "inteligencia artificial"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "Python"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "12/25/2023"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
 
word = "😊"
tokens, decode_tokens = encode_decode(word)
print(f"Word: {word} ==&gt; tokens: {tokens}, decode_tokens: {decode_tokens}")
	
	Copied

>_ Output

			
				Word: perro ==&gt; tokens: [716, 299], decode_tokens: ['per', 'ro']
Word: perra ==&gt; tokens: [79, 14210], decode_tokens: ['p', 'erra']
Word: mañana... ==&gt; tokens: [1764, 88184, 1131], decode_tokens: ['ma', 'ñana', '...']
Word: inteligencia artificial ==&gt; tokens: [396, 39567, 8968, 21075], decode_tokens: ['int', 'elig', 'encia', ' artificial']
Word: Python ==&gt; tokens: [31380], decode_tokens: ['Python']
Word: 12/25/2023 ==&gt; tokens: [717, 14, 914, 14, 2366, 18], decode_tokens: ['12', '/', '25', '/', '202', '3']
Word: 😊 ==&gt; tokens: [76460, 232], decode_tokens: ['�', '�']

We can see that for similar words, Spanish generates more tokens than English, so for the same text, with a similar number of words, the number of tokens will be greater in Spanish than in English.

Frequently asked questions

Why does decoding an emoji's token with tiktoken show the ? character instead of the emoji?

Because tiktoken (using the cl100k_base encoding) operates at the byte level, and an emoji like 😊 spans several UTF-8 bytes that get split across two tokens ([76460, 232] in the post's example). When you decode each token separately with encoder.decode([token]), each one only represents part of the emoji's byte sequence, which isn't a valid standalone UTF-8 character, so Python renders it as the replacement character ?. Only decoding both tokens together reconstructs the original emoji.

Why does the same piece of text need more tokens in Spanish than in English with OpenAI's tokenizer?

Because tiktoken's vocabulary (100,277 tokens for cl100k_base, per encoder.n_vocab) is built mostly from English text, so common English words tend to land in a single token while their Spanish equivalents get split into more pieces. The post shows "dog" → 1 token (['dog']) versus "perro" → 2 tokens (['per', 'ro']), or "artificial intelligence" (3 tokens) versus "inteligencia artificial" (4 tokens). The upshot: for roughly the same number of words, a Spanish text burns through more tokens than its English equivalent.

Continue reading

Hugging Face Tokenizers: Fast Encoding

Guide to Hugging Face Tokenizers library: transform text into structured data for AI models. Practical Python examples for efficient NLP tokenization.

OpenAI API: Integration & Practical Guide

Complete OpenAI API tutorial: learn to use GPT for text generation, DALL-E for images, and more. Setup, installation, an...

BPE Tokenizer: Subword Encoding Algorithm

Understand BPE (Byte Pair Encoding) tokenization for NLP. Learn how this popular method splits text into tokens with ste...

Last posts -->

Have you seen these projects?

Gymnasia

LangGraph Deep Researcher

Tau

View all projects -->

>_ Available for projects

Do you have an AI project?

Let's talk.

maximofn@gmail.com

Machine Learning and AI specialist. I develop solutions with generative AI, intelligent agents and custom models.

Write me LinkedIn

Do you want to watch any talk?

Tomorrow's Agents: Deciphering the Mysteries of Planning, UX and Memory

AI agents, powered by LLMs, promise to transform applications. But are they simple executors today or future intelligent collaborators? To reach their...

Create your own Apple intelligence

Learn to create an IA system to execute efficiently on a device

Last talks -->

Do you want to improve with these tips?

Best practices building agents with Claude Code

Technical talk: skills, subagents, slash commands and MCPs in Claude Code

o1 prompt engineering

Create better prompts for o1 following an example

Memory profiler

See the memory usage of a script

Last tips -->

Use this locally

Hugging Face spaces allow us to run models with very simple demos, but what if the demo breaks? Or if the user deletes it? That's why I've created docker containers with some interesting spaces, to be able to use them locally, whatever happens. In fact, if you click on any project view button, it may take you to a space that doesn't work.

Flow edit

FLUX.1-RealismLora

token_hmr

View all containers -->

>_ Available for projects

Do you have an AI project?

Let's talk.

maximofn@gmail.com

Machine Learning and AI specialist. I develop solutions with generative AI, intelligent agents and custom models.

Write me LinkedIn

Do you want to train your model with these datasets?

short-jokes-dataset

HuggingFace

Dataset with jokes in English

Use: Fine-tuning text generation models for humor

231K rows 2 columns 45 MB

View on HuggingFace →

opus100

HuggingFace

Dataset with translations from English to Spanish

Use: Training English-Spanish translation models

1M rows 2 columns 210 MB

View on HuggingFace →

netflix_titles

HuggingFace

Dataset with Netflix movies and series

Use: Netflix catalog analysis and recommendation systems

8.8K rows 12 columns 3.5 MB

View on HuggingFace →

View more datasets -->