Editorial illustration for Gigatoken BPE Encoder Hits 24.53 GB/s, Up to 989x Faster Than HuggingFace
Gigatoken BPE Encoder Hits 989x Speed Over HuggingFace
Marcel Rød, a PhD student at Stanford, released a Rust tokenizer called Gigatoken under an MIT license, and the benchmark numbers attached to it are hard to ignore. On an 11.9 GB slice of the owt_train.txt corpus, run on a dual-socket AMD EPYC 9565 with 144 cores, Gigatoken's GPT-2 tokenizer hit 24.53 GB/s. HuggingFace's tokenizers library, on the same machine, managed 24.8 MB/s.
OpenAI's tiktoken did 36.0 MB/s. That's not a rounding error in someone's favor, it's a gap of roughly three orders of magnitude, and it shows up again on an Apple M4 Max (8.79 GB/s) and a Ryzen 7 9800X3D (6.27 GB/s), against baselines already written in multithreaded Rust. Tokenization tends to get treated as plumbing, the part of the pipeline nobody bothers to profile because the model itself gets all the attention.
Gigatoken's numbers, spanning 23 tokenizer families from GPT-OSS to Llama 4 to DeepSeek V3, suggest that assumption has been costing real throughput. The library ships on PyPI as version 0.9.0, installable with a single pip command, built mostly in Rust with Python bindings on top.
The GPT-2 tokenizer benchmarking yields remarkable results: evaluated on the 11.9 GB owt_train.txt corpus using a 144-core AMD EPYC 9565 dual-socket setup, Gigatoken processes data at a staggering 24.53 GB/s. In comparison, OpenAI’s tiktoken achieves 36.0 MB/s, while HuggingFace tokenizers registers at 24.8 MB/s on the identical hardware configuration. These marks demonstrate performance advantages of 681x and 989x, respectively.
Why this matters
Tokenization has been the part of the pipeline everyone assumes is already solved, mostly because tiktoken and HuggingFace's tokenizers are fast enough not to complain about. Gigatoken's numbers suggest "fast enough" left a lot on the table. An 83.4x gap over tokenizers 0.23.1 on a 4-vCPU box, and 989x claimed on the full EPYC run, isn't a rounding error in a benchmark table.
It's the kind of gap that changes how you think about preprocessing budgets for large corpora. For anyone training on tens or hundreds of gigabytes of text, tokenization throughput at that scale can eat real wall-clock time and real compute spend, especially on rented CPU instances where every hour counts. We'd want to see these numbers replicated outside Rød's own test setup, on different corpora and hardware, before treating 24.53 GB/s as a stable baseline rather than a best case.
But MIT-licensed and reproducible on commodity hardware, this is worth a look for anyone still running data pipelines through the default tokenizer library without asking why that step takes as long as it does.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Naval Postgraduate School Activates NVIDIA AI Supercomputer for In-House Training
From Archives Britain's AI safety tests find models 'cheating' on cybersecurity evaluations
Common Questions Answered
How much faster is Gigatoken's GPT-2 tokenizer compared to HuggingFace's tokenizers library?
Gigatoken achieves 24.53 GB/s on the owt_train.txt corpus, while HuggingFace's tokenizers library manages only 24.8 MB/s on the same hardware, representing a performance advantage of 989x. This massive gap demonstrates that Gigatoken is nearly 1000 times faster than HuggingFace's solution for tokenization tasks.
What hardware configuration was used to benchmark Gigatoken's tokenization speed?
The benchmarks were conducted on a dual-socket AMD EPYC 9565 processor with 144 cores, processing an 11.9 GB slice of the owt_train.txt corpus. This high-end server hardware configuration allowed Gigatoken to demonstrate its peak performance capabilities of 24.53 GB/s.
Who developed Gigatoken and what programming language was it written in?
Marcel Rød, a PhD student at Stanford, developed Gigatoken as a Rust-based tokenizer and released it under an MIT license. The choice of Rust as the implementation language likely contributes to Gigatoken's exceptional performance compared to other tokenizer libraries.
Why does Gigatoken's performance improvement matter for large language model preprocessing?
Gigatoken's 989x speed advantage over existing solutions fundamentally changes how practitioners should think about preprocessing budgets for large corpora, as tokenization is no longer a bottleneck that can be ignored. The massive performance gap suggests that organizations can significantly reduce their data preprocessing time and computational costs when handling large-scale datasets for model training.
How does Gigatoken's performance compare to OpenAI's tiktoken?
Gigatoken processes data at 24.53 GB/s while OpenAI's tiktoken achieves 36.0 MB/s on the same AMD EPYC 9565 hardware, giving Gigatoken a 681x performance advantage. This demonstrates that Gigatoken substantially outperforms even OpenAI's optimized tokenizer implementation.
Further Reading
- Marcel Røed Releases Gigatoken on GitHub - AI Weekly
- Marcel Roed releases Rust tokenizer 100x faster than ... - Digg
- BlockBPE: Parallel BPE Tokenization - arXiv
- Building a Fast BPE Tokenizer from Scratch - JY Tan
- Byte-Pair Encoding tokenization - Hugging Face
