$ tokenizers
v0.22.2Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Downloads: 8.1M
Recent: 3.0M
Versions: 39
Updated: December 2, 2025
Patch update: 0.22.1 → 0.22.2
$ DOWNLOADS TREND
6.1M8.1M
$ VERSION HISTORY
v0.22.2December 2, 2025
v0.22.1September 19, 2025
$ LINKS
$ INSTALL
cargo add tokenizersOr add to Cargo.toml: tokenizers = "0.22.2"