In early 2024, this simple question stumped GPT-4. A billion-dollar AI model failed at counting letters in a 10-letter word. Why?

The answer lies in tokenization.

🎯 More Tokenization Failures

Why Tokenization Matters: Three Real-World Impacts

1. 💰 Cost

API pricing is typically based on token usage – the more tokens, the higher the cost. Token efficiency directly translates to money.

For a company making 1 million API calls per day with an average of 50 tokens per call:

• With GPT-2 efficiency: 50M tokens/day x $0.03/1K = $1,500/day = $547,500/year
• With GPT-4 efficiency: 35M tokens/day x $0.03/1K = $1,050/day = $383,250/year
• Savings: $164,250 per year just from tokenization efficiency!

2. 🌍 Fairness

Tokenizers trained primarily on English text penalize users of other languages. This isn’t just unfair – it’s a structural bias in how LLMs work.

This affects three critical areas:

• API costs: Non-English speakers literally pay 2-3x more for the same functionality
• Context window: Less actual content fits for non-English users (8K tokens = less text)
• Model performance: Fewer tokens means the model “sees” less context, degrading quality

3. 🎯 Capability

What the tokenizer can “see” fundamentally determines what the model can learn. This is why specialized models exist.

StarCoder2’s tokenizer has:

• Single tokens for Python keywords (def, class, return)
• Single tokens for multiple spaces (handles Python indentation properly)
• Digit-by-digit number tokenization (better mathematical reasoning)
• Special tokens for code structure

How Tokenization Works

Now that you understand why tokenization matters, let’s explore how it actually works. When you type text into ChatGPT, a crucial transformation happens before the model even begins to “think.”

Think of it like this: If you’re trying to understand a book, but someone only gives you whole pages at a time (never individual words or sentences), your understanding would be limited. Similarly, if someone gives you individual letters, you’d spend all your time assembling them into words. Tokens are the “just right” level of granularity.

Let’s See It In Action

Here’s how to tokenize “strawberry” and see why GPT-4 can’t count the ‘r’s:

The model never sees “s-t-r-a-w-b-e-r-r-y”. It sees two tokens: 2536 and 19772. Those are just numbers referencing entries in the tokenizer’s vocabulary.

The Four Tokenization Paradigms

There are fundamentally different approaches to breaking down text. Modern LLMs use subword tokenization, but understanding all four paradigms reveals why.

Why subword tokenization wins: It’s the “Goldilocks” solution. Common words stay intact (“AI”, “is”), while rare words break into reusable pieces (“amaz” + “ing”). This allows the model to understand new words it’s never seen before.

Comparing Real Tokenizers: The Evolution (2018-2024)

Over six years, tokenizers went from losing information ([UNK] tokens) to being 35% more efficient while handling emojis, multiple languages, and specialized domains like code.

Design Trade-offs Revealed

Looking at this comparison, several fundamental trade-offs become clear:

Vocabulary size vs. Model size: GPT-4’s ~100K vocabulary makes tokenization more efficient, but the embedding layer (which maps token IDs to vectors) becomes larger. A 30K vocabulary needs 30K vectors; a 100K vocabulary needs 100K vectors. This is pure memory.

General vs. Specialized: GPT-4 optimizes for general text across many languages. StarCoder2 sacrifices some text efficiency (digit-by-digit numbers) to excel at code. You can’t optimize for everything simultaneously.

Efficiency vs. Capability: StarCoder2’s digit-by-digit tokenization uses more tokens (lower efficiency, higher cost) but enables better mathematical reasoning. When “870” and “871” tokenize as [8][7][0] and [8][7][1], the model sees the pattern that they differ by just one digit.

Language fairness vs. Optimization: To make English tokenization efficient, you need English-heavy training data for the tokenizer. This inevitably makes other languages less efficient. There’s no way around this with current approaches.

Try It Yourself – Complete Code

Here’s clean, runnable Python code to compare all tokenizers:

Installation:

Expected output:

Practical Takeaways for LLM Users

When Choosing a Model

• Test your actual content: Don’t assume token counts. Use tiktoken or the OpenAI tokenizer playground to see how YOUR prompts tokenize.
• Consider domain-specific models: Need code? StarCoder beats GPT-4. Need science? Galactica might be better. General purpose? GPT-4/Claude are optimal.
• Account for language bias: If you’re building for non-English users, factor in 2-3x token inflation in your cost estimates.

When Writing Prompts

• Be concise: Every word costs tokens. “Please explain” vs “Explain” is one extra token per request. Multiply by millions of requests.
• Avoid unnecessary formatting: Markdown bullets, headers, and emphasis all cost tokens. Use them only when they improve output quality.
• Front-load important context: Models pay more “attention” to recent tokens. Put critical information near the end of your prompt.

When Building Applications

• Monitor token usage: Track average tokens per request, per user, per feature. Find your inefficiencies.
• Implement token budgets: Limit context window usage. Don’t send entire documents when a summary would work.
• Cache tokenized inputs: If you’re sending the same system prompt on every request, cache the tokenized version.
• Chunk strategically: When splitting long documents, split at semantic boundaries (paragraphs, sections), not arbitrary token counts.
• Use traditional code for letter-level tasks: Don’t ask an LLM to count letters, reverse strings, or do arithmetic. Use Python for that.

Understanding the Limitations

Knowing how tokenization works helps you understand when LLMs will struggle:

• Letter-level tasks: Counting, reversing, anagrams – the model can’t see individual letters
• Exact arithmetic: Inconsistent number tokenization makes patterns hard to learn
• Spelling: The model can spell common words (they’re single tokens) but struggles with rare words
• Character-level patterns: Detecting repeated characters, palindromes, etc.

For these tasks, use traditional programming. For everything else – pattern matching, generation, understanding context – LLMs excel precisely because they work at the token level.

What’s Next?

Now that you understand tokenization – the first step in how LLMs process text – you’re ready for the next pieces of the puzzle:

• Embeddings: How do those token IDs become meaningful numeric vectors? What makes “king” – “man” + “woman” approximately equal “queen” work mathematically?
• The Transformer architecture: What happens inside the model after tokenization? How does self-attention work?
• Training dynamics: How do models learn from tokens? What’s the role of the tokenizer during training vs. inference?
• Advanced tokenization: BPE vs. WordPiece vs. Unigram – what are the differences? How are tokenizers trained?

These topics will be covered in future posts on DataHacker.rs. Understanding tokenization first makes everything else make more sense.

Key Takeaways