|

Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

ByRicardo

Tokenization is the one a part of the language modeling stack that nearly no person profiles. Gigatoken, launched by Marcel Rød (a PhD pupil from Stanford)  underneath an MIT license, argues that this was a mistake. The library encodes textual content at gigabytes per second on a single machine, in opposition to baselines that are already multithreaded Rust.

The GPT-2 tokenizer benchmarking yields exceptional outcomes: evaluated on the 11.9 GB owt_train.txt corpus utilizing a 144-core AMD EPYC 9565 dual-socket setup, Gigatoken processes knowledge at a staggering 24.53 GB/s. In comparability, OpenAI’s tiktoken achieves 36.0 MB/s, whereas HuggingFace tokenizers registers at 24.8 MB/s on the an identical {hardware} configuration. These marks exhibit efficiency benefits of 681x and 989x, respectively.

On an Apple M4 Max with 16 cores, the identical GPT-2 workload runs at 8.79 GB/s, or 1,268x HuggingFace tokenizers and 140x tiktoken. On a shopper AMD Ryzen 7 9800X3D, it runs at 6.27 GB/s, or 106x and 68x. The speedup is just not an artifact of 1 CPU or one vocabulary.


How it will get there

The beneficial properties don’t come from a greater BPE merge loop. They come from two locations that most tokenizers deal with as solved.

(1) Pretokenization: Most implementations delegate this to a regex engine. Gigatoken hand-writes it. The optimization log tracks single-threaded GPT-2 pretokenizer throughput on 100 MB of OpenWebText, and the development is instructive:

  • A fancy-regex baseline runs at ~47 MiB/s. A hand-rolled state machine hits ~380 MiB/s. A winnow-combinator implementation with NEON SIMD intrinsics reaches 462 MiB/s.
  • Replacing winnow with a direct Iterator, including a 256-byte class lookup desk for O(1) first-byte dispatch, and switching from NEON intrinsics to SWAR (SIMD Within A Register) pushes it to 830 MiB/s. SWAR masses 8 bytes as a u64 and checks all 8 for the letter property with branchless arithmetic. It wants no architecture-specific intrinsics.
  • The last step is dual-cursor ILP exploitation, reaching 1,049 MiB/s. The perception is that the bottleneck at ~840 MiB/s was latency, not throughput. Each token’s finish place relies on the earlier one, a serial chain of roughly 25–27 cycles. Running two unbiased cursors from a protected cut up level lets the out-of-order engine interleave each streams throughout idle execution ports.

Net impact on the pretokenizer alone: 2.27x over the winnow + NEON baseline, and 22.3x over the regex implementation.

(2) Pretoken caching: If a phrase has been seen earlier than, its encoded tokens are seemed up fairly than recomputed. The writer (Marcel) notes that is onerous in apply, as a result of the cache grows shortly and pretoken distributions are long-tailed. On high of that, interactions with Python are minimized and threads are designed to work together minimally with one another.

The optimization log is unusually trustworthy about what failed. A sizzling/chilly cut up utilizing #[cold] and #[inline(never)] regressed to 580 MiB/s and was reverted, as a result of the inline barrier stopped LLVM from optimizing the mixed ASCII and unicode loop. A two-pass classification buffer with SWAR transition counting was algorithmically right however ran at 354 MiB/s, for the reason that additional reminiscence site visitors beat the department financial savings. Profile-guided optimization had no measurable impact, as a result of the inside loop is already branchless and the word-boundary department is data-dependent.

Benchmark Methodology Notes

The comparability is just not strictly apples-to-apples. Gigatoken encodes whole un-split recordsdata, discovering its personal doc boundaries and parallelizing mechanically. In distinction, HuggingFace (encode_batch_fast) is evaluated on the primary 100 MB and tiktoken (encode_ordinary_batch) on the primary 1 GB, each pre-split on <|endoftext|>. Because the baselines omit caching, their throughput stays uniform. Measurements report one of the best of three interleaved rounds utilizing recent processes with parallelism enabled.

Vocabulary varieties additionally introduce constraints; SentencePiece tokenizers are solely partially optimized. On EPYC, Gemma 1 processes at 2.51 GB/s (7.3x speedup), Gemma 3 at 3.43 GB/s (9.6x), and CodeLlama at 3.47 GB/s (10.0x). Although substantial, these beneficial properties are an order of magnitude decrease than the headline BPE efficiency.

An unbiased replica on KrabArena verified the outcomes. On a 4-vCPU Intel Xeon VM (2.20 GHz) with a 174 MB OpenWebText slice, Gigatoken 0.9.0 achieved a median of 277.8 MB/s, outperforming tiktoken 0.13.0 (10.62 MB/s) by 26.2x and tokenizers 0.23.1 (3.33 MB/s) by 83.4x. All trials efficiently validated 35,356 paperwork, confirming the efficiency development scales with core depend.

Key Takeaways

  • Gigatoken hits 24.53 GB/s on GPT-2 on a 144-core EPYC, 989x HuggingFace tokenizers and 681x tiktoken.
  • The speedup holds throughout x86 and ARM, and throughout 23 tokenizer households, not one tuned configuration.
  • Gains come from a hand-written SWAR pretokenizer and pretoken caching, not a sooner BPE merge loop.
  • SentencePiece vocabularies see 7–22x, not 1,000x, and WordPiece is unsupported.
  • Compatibility mode preserves precise HuggingFace output parity at roughly 200–300x as a substitute of the complete speedup.

Check out the GitHub repo and the release thread. All credit score for this analysis goes to the writer of this undertaking.

The publish Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers appeared first on MarkTechPost.

Similar Posts