Hacker GitHub

← All repos

alasdairforsythe/tokenmonster

View on GitHub ↗ ★ 625 Go

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

4 mentions on Hacker News by 4 people: 2 stories, 2 comments · first 2023-06-04 · latest 2026-04-20

Mentions per year 4 mentions · busiest year 2023 (3)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 1
    Tokenmonster: Determine tokens to optimally represents a dataset
    story by anotherpaulg · 2023-06-09 · 0 comments
  2. ▲ 10
    TokenMonster: Ungreedy tokenizer, outperforming tiktoken by 35%
    story by tosh · 2023-06-04 · 0 comments