Hacker GitHub

← All repos

openai/tiktoken

View on GitHub ↗ ★ 19,581 Python

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

28 mentions on Hacker News by 26 people: 1 story, 27 comments · first 2022-12-16 · latest 2026-09-05

Mentions per year 28 mentions · busiest year 2023 (10)
20082026
AllStoriesComments NewestTopOldest
  1. …count though! I just checked the tiktoken library and couldn't see any changes relating to Luna: https://github.com/openai/tiktoken
    comment by simonw · 2026-09-05 · in GPT-6 Astra on OpenRouter
  2. …symbol ∅ can’t possibly have appeared that often outside of things like set theory and logic. [1] - https://github.com/openai/tiktoken
    comment by vunderba · 2026-08-11 · in Show HN: Mcptoon – Token-efficient MCP CLI client
  3. …by the tokenization of the training and input data? See https://platform.openai.com/tokenizer https://github.com/openai/tiktoken
    comment by amai · 2025-08-08 · in GPT-5: "How many times does the letter b appear in blueberry?"
  4. …is increasingly adopting Rust (at least for inference). [1] https://github.com/openai/harmony [2] https://github.com/openai/tiktoken [3] https://github.com/openai/codex
    comment by lukax · 2025-08-05 · in Open models by OpenAI
  5. There is at least some awareness already when it comes to the performance of the regex engine: https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-...
    comment by 22c · 2025-07-02 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
  6. It looks like TikToken is written in Rust ( https://github.com/openai/tiktoken/tree/main/src ), are the gains here actually from porting to C++?
    comment by notatallshaw · 2025-06-30 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
  7. …considered [1]. So it's faster to check against the much smaller list we _know_ is being used. [0] https://github.com/openai/tiktoken/blob/main/src/lib.rs#L476 [1] https://github.com/openai/tiktoken/blob/main/tiktoken/core.p...
    comment by matthewolfe · 2025-06-30 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
  8. …so you shouldn't see more of those connections, if I'm understanding the code correctly. [0] < https://github.com/openai/tiktoken >
    comment by degamad · 2025-05-14 · in PDF to Text, a challenging problem
  9. …went back to SentencePiece in the first place? Byte-pair encoding (which is what tiktoken uses: https://github.com/openai/tiktoken ) was shown to be a more efficient encoding as far back as GPT-2 in 2019.
    comment by minimaxir · 2024-07-18 · in Mistral NeMo
  10. New tokenizer has a much larger vocabulary (200k)[0]. [0] https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74c...
    comment by zackangelo · 2024-05-13 · in GPT-4o
  11. Tiktoken added support for GPT-4o: https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74c... It has an increased vocab size of 200k.
    comment by atgctg · 2024-05-13 · in GPT-4o
  12. …are in fact in rust, with an interface in python: https://github.com/huggingface/tokenizers https://github.com/openai/tiktoken
    comment by m3at · 2024-04-07 · in Tokens, n-grams, and bag-of-words models (2023)
  13. Yeah, see [1]. [1] https://github.com/openai/tiktoken/blob/main/tiktoken/model....
    comment by lifthrasiir · 2024-01-22 · in GPT-3.5 crashes when it thinks about useRalativeImagePath too much
  14. Their tokenizer is open source: https://github.com/openai/tiktoken Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...
    comment by dchest · 2024-01-22 · in GPT-3.5 crashes when it thinks about useRalativeImagePath too much
  15. …them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken In theory they could be added in normal input but it's possible OpenAI has safeguards…
    comment by minimaxir · 2023-11-21 · in Claude 2.1
  16. https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
    comment by sp332 · 2023-06-13 · in Show HN: LLaMA tokenizer that runs in browser
  17. …list on my computer shows up in /tmp/data-gym-cache/9b5ad71b2ce5302211f9c61530b329a4922fc6a4 [1] https://github.com/openai/tiktoken
    comment by s-macke · 2023-04-23 · in List of all 100k GPT-4 tokens
  18. …nice job of explaining it [2]. Plus the original paper has a very readable Python example [3]. [1] https://github.com/openai/tiktoken [2] https://huggingface.co/course/chapter6/5?fw=pt [3] https://arxiv.org/abs/1508.07909
    comment by montebicyclelo · 2023-04-15 · in What are transformer models and how do they work?
  19. …(unclear how well it performs for non-English languages) You can test it offline using tiktoken: https://github.com/openai/tiktoken
    comment by minimaxir · 2023-04-10 · in Why is GPT-3 15.77x more expensive for certain languages?
  20. If this splitting interest you having a look at https://github.com/openai/tiktoken/blob/main/src/lib.rs is great at showing all the ugly edge cases that causes instabilities. In theory Byte Pair Encoding is unique, but practice makes it…
    comment by GistNoesis · 2023-04-06 · in OpenAI Tokenizer
  21. …bit out of date as GPT4 has a different tokenizer than GPT3. I'd recommend checking out tiktoken ( https://github.com/openai/tiktoken ) or this other excellent app that a community member made ( https://tiktokenizer.vercel.app )
    comment by dschnurr · 2023-04-05 · in OpenAI Tokenizer
  22. https://github.com/openai/tiktoken
    comment by sharkjacobs · 2023-04-05 · in OpenAI Tokenizer
  23. You can use https://github.com/openai/tiktoken
    comment by iamjackg · 2023-04-05 · in OpenAI Tokenizer
  24. OpenAI seems to use Tiktoken [0]. It also covers GPT-4 token encoding. [0] https://github.com/openai/tiktoken
    comment by coffeeri · 2023-04-05 · in OpenAI Tokenizer
  25. …this further? Seems like there is significant low hanging fruit, as evidenced by this line: https://github.com/openai/tiktoken/blob/main/src/lib.rs#L419
    comment by FL33TW00D · 2022-12-17 · in Tiktoken: OpenAI’s Tokenizer
  26. …of ML/AI stuff wants to your sign some sort of license What's the point of this MIT license, then: https://github.com/openai/tiktoken/blob/main/LICENSE
    comment by yencabulator · 2022-12-16 · in Tiktoken: OpenAI’s Tokenizer
  27. https://github.com/openai/tiktoken/blob/main/tests/test_simp...
    comment by stabbles · 2022-12-16 · in Tiktoken: OpenAI’s Tokenizer
  28. ▲ 153
    Tiktoken: OpenAI’s Tokenizer
    story by azhenley · 2022-12-16 · 74 comments