Learn AI / Tools / Tokenization visualizer
BPE tokenization visualizer.
Byte pair encoding is how most language models turn text into tokens. Type anything, then step through the algorithm as it counts adjacent pairs, merges the most common one, and grows its vocabulary.
Step 0 of 0
Tokenization rust outline = pair about to merge · espresso outline = token created this step
Pair frequencies
| Pair | Merged | Count |
|---|
What is this?
Each step counts every adjacent pair of tokens in the sequence. The most frequent pair (top row, highlighted) is merged into a single new token wherever it appears. Ties are broken by first appearance. The algorithm stops when no pair occurs more than once.BPE steps click a step to jump
Vocabulary statistics
0unique tokens in vocabulary
1.00compression ratio (characters ÷ tokens)
0tokens in sequence
0merges applied so far