Hacker GitHub

← All repos

alasdairforsythe/tokenmonster

View on GitHub ↗ ★ 625 Go

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

4 mentions on Hacker News by 4 people: 2 stories, 2 comments · first 2023-06-04 · latest 2026-04-20

Mentions per year 4 mentions · busiest year 2023 (3)
20082026
AllStoriesComments NewestTopOldest
  1. …Haiku also show the Title Case token (↑). This is similar to what the TokenMonster tokenizer does: https://github.com/alasdairforsythe/tokenmonster
    comment by yorwba · 2026-04-20 · in Claude Token Counter, now with model comparisons
  2. The actual analysis: https://github.com/alasdairforsythe/tokenmonster/blob/main/b... > Summary of Findings: > - Comparable (50256-strict-nocapcode) TokenMonster vocabularies perform better than both GPT-2 Tokenizer and tiktoken p50k_base…
    comment by alecco · 2023-09-05 · in Tokenizer benchmark comparing 16 language models pre-trained from scratch
  3. ▲ 1
    Tokenmonster: Determine tokens to optimally represents a dataset
    story by anotherpaulg · 2023-06-09 · 0 comments
  4. ▲ 10
    TokenMonster: Ungreedy tokenizer, outperforming tiktoken by 35%
    story by tosh · 2023-06-04 · 0 comments