Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
Tokenization is the one a part of the language modeling stack that nearly no person profiles. Gigatoken, launched by Marcel Rød (a PhD pupil from Stanford) underneath an MIT license, argues that this was a mistake. The library encodes textual content at gigabytes per second on a single machine, in opposition to baselines that are…
