A Deep Dive into Tokenizers: How AI Reads Text

Large language models (LLMs) do not read whole words or sentences the way humans do. Instead, an AI relies on a tokenizer to chop sentences into bite-sized chunks called tokens.

To see how tokenizers work under the hood, look at the industry's go-to method: Byte Pair Encoding (BPE).

How Byte Pair Encoding (BPE) Works

Think of BPE like building with Lego bricks. It starts with the smallest individual pieces and snaps frequent pairs together to make larger blocks.

• Step 1: Start with basic characters. The algorithm begins with a base vocabulary containing individual letters (A–Z, a–z), numbers, and punctuation marks.
• Step 2: Merge the most frequent pairs. As the model scans massive amounts of training text, it looks for character pairs that appear side-by-side all the time and welds them into a single token (e.g., t + h = th).
• Step 3: Build longer sequences. Once two-character pairs are mapped, the algorithm merges those tokens into longer frequent chunks (e.g., th + e = the).

Most Common Character Patterns in English

Vocabulary Limits: OpenAI’s o200k_base

Every AI model caps its total dictionary size to keep memory manageable.

• Vocabulary Size: OpenAI's modern models (like GPT-4o) use the o200k_base tokenizer, which caps the vocabulary at around 200,000 unique tokens. The model can only read or generate text using chunks from this preset catalog.
• Token Length Limit: A single token cannot be infinitely long. In o200k_base, the maximum length of an individual token is 256 bytes.
• Real-World Impact: In standard English, 1 byte roughly equals 1 character. That means the longest single token can stretch up to 256 characters—useful for compressing repetitive code structures, like a row of dashes --- inside Markdown files.