huggingface/tokenizers
View on GitHub ↗ ★ 11,230 Rust
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
6 mentions on Hacker News by 5 people: 1 story, 5 comments · first 2020-01-13 · latest 2025-06-30
20082026
-
Can you also compare the performance with https://github.com/huggingface/tokenizers/ ? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.
comment by Tiberium · 2025-06-30 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken -
…libraries for token creation (tokenization) are in fact in rust, with an interface in python: https://github.com/huggingface/tokenizers https://github.com/openai/tiktoken
comment by m3at · 2024-04-07 · in Tokens, n-grams, and bag-of-words models (2023) -
Huggingface Tokenizers ( https://github.com/huggingface/tokenizers ), which are now used by default in their Transformers Python library, use pyO3 and became popular due to the pitch that it encoded text an order of magnitude faster with…
comment by minimaxir · 2021-01-29 · in PyO3: Rust Bindings for the Python Interpreter -
…some raw text as input and outputs an Encoding. The various steps of the pipeline are: ...." [1] https://github.com/huggingface/tokenizers#quick-examples-usi... [2] https://github.com/huggingface/tokenizers/tree/master/tokeni...
comment by wyldfire · 2020-01-13 · in Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines -
…token masks...) - Written in Rust with bindings for Python and node.js Github repository and doc: https://github.com/huggingface/tokenizers/tree/master/tokeni... To install: - Rust: https://crates.io/crates/tokenizers - Python: pip install…
comment by julien_c · 2020-01-13 · in Show HN: HuggingFace – Fast tokenization library for deep-learning NLP pipelines