Hacker GitHub

← All repos

google/sentencepiece

View on GitHub ↗ ★ 12,163 C++

Unsupervised text tokenizer for Neural Network-based text generation.

11 mentions on Hacker News by 11 people: 1 story, 10 comments · first 2018-09-06 · latest 2025-06-30

Mentions per year 11 mentions · busiest year 2023 (5)
20082026
AllStoriesComments NewestTopOldest
  1. …labs (OpenAI, Anthropic, Google), only Anthropic with Claude 3 and newer lack local tokenizers. [1] https://github.com/google/sentencepiece [2] https://github.com/googleapis/python-aiplatform/blob/main/ve... [3]…
    comment by Deathmax · 2025-06-30 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
  2. I thought Gemini used SentencePiece https://github.com/google/sentencepiece
    comment by weberer · 2025-06-30 · in Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
  3. …whitespace boundaries by setting flag `--split-on-whitespace` to false (it's true by default). https://github.com/google/sentencepiece/blob/master/doc/opti...
    comment by mcyc · 2025-05-31 · in Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
  4. …Including the frequency weighting is a bit nonstandard among ablative methods. See line 233: https://github.com/google/sentencepiece/blob/master/src/unig... I would suspect the n-gram counts don't cross pre-token boundaries, but I don't…
    comment by cschmidt · 2025-05-31 · in Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
  5. …trained). you need to train the model on 1 trillion tokens ( https://platform.openai.com/tokenizer https://github.com/google/sentencepiece ) anyways for it to get reasoning capacities, which it feels very unlikely that your data is that…
    comment by make3 · 2023-06-13 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  6. The original llama uses google's tooling for that, written in C++ https://github.com/google/sentencepiece
    comment by Const-me · 2023-03-12 · in Large language models are having their Stable Diffusion moment
  7. …slow as I focused on getting it to run, not making it fast. The tokenizer uses some Google thing https://github.com/google/sentencepiece but luckily for inference it seems that you just need to be able to parse the tokenizer model file and…
    comment by adeon · 2023-03-11 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
  8. …for dates? [1]: http://nlpprogress.com/english/intent_detection_slot_filling... [2]: https://github.com/google/sentencepiece
    comment by jldugger · 2023-01-22 · in ChatGPT in an iOS Shortcut: Smart HomeKit Voice Assistant
  9. …like sentencepiece with sub-word splitting and then charge by the number of resulting token. https://github.com/google/sentencepiece
    comment by belval · 2022-04-06 · in Dall-E 2
  10. …with Subword Units https://arxiv.org/abs/1508.07909a A Python library for BPE ngrams: sentencepiece https://github.com/google/sentencepiece
    comment by visarga · 2018-09-06 · in Fast word vectors with little memory usage in Python