clipperhouse/uax29
View on GitHub ↗ ★ 125 Go
A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.
8 mentions on Hacker News by 1 person: 3 stories, 5 comments · first 2020-04-22 · latest 2026-01-05
20082026
-
…boundaries, but also thought that the notion of a “phrase” might be useful for such applications: https://github.com/clipperhouse/uax29/tree/master/phrases
comment by mwsherman · 2026-01-05 · in So, you want to chunk really fast? -
…it’s a Unicode thing, not a Go thing. Shameless plug, here is a grapheme tokenizer for Go: https://github.com/clipperhouse/uax29/tree/master/graphemes
comment by mwsherman · 2025-10-22 · in Go subtleties -
Shameless plug, you may wish to do Lucene-style tokenizing using the Unicode standard: https://github.com/clipperhouse/uax29/tree/master/words
comment by mwsherman · 2025-10-09 · in Show HN: I wrote a full text search engine in Go -
…was Spans. Notably, it caused me to realize that Go had “spans” designed in from the start. [1] https://github.com/clipperhouse/uax29
comment by mwsherman · 2025-09-30 · in Safe zero-copy operations in C# -
…Lucene tokenizer uses it. I implemented it for Go[1]. [0] https://unicode.org/reports/tr29/ [1] https://github.com/clipperhouse/uax29
comment by mwsherman · 2020-08-05 · in Let's build a full-text search engine