Hacker GitHub

← All repos

simonw/ttok

View on GitHub ↗ ★ 409 Python

Count and truncate text based on tokens

3 mentions on Hacker News by 3 people: 0 stories, 3 comments · first 2023-09-22 · latest 2024-09-26

Mentions per year 3 mentions · busiest year 2024 (2)
20082026
AllStoriesComments NewestTopOldest
  1. I use my https://github.com/simonw/ttok command for that - you can pipe stuff into it for a token count. Unfortunately it only uses the OpenAI tokenizers at the moment (via tiktoken), so counts for other models may be inaccurate. I find…
    comment by simonw · 2024-09-26 · in Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
  2. …for machine learning models, especially in NLP applications. - *Link*: [ttok GitHub Repository]( https://github.com/simonw/ttok ) ### 3. symbex `Symbex` is a CLI tool for performing symbolic execution on Python bytecode. It can analyze…
    comment by irthomasthomas · 2024-05-19 · in Show HN: Interactive Graph by LLM (GPT-4o)
  3. …it was a tad over 16k. So what does your site do in that case? Is the transcript truncated? [0]: https://github.com/simonw/ttok
    comment by gkfasdfasdf · 2023-09-22 · in Show HN: Summary Cat, a YouTube Video Summary Generator