fxsjy/jieba
View on GitHub ↗ ★ 35,227 Python
结巴中文分词
4 mentions on Hacker News by 3 people: 0 stories, 4 comments · first 2018-09-05 · latest 2026-04-17
20082026
-
…2. Split into sentences and tokenize sentences into words, e.g. using https://github.com/fxsjy/jieba 3. Count how often each word appears and sort sentences by descending frequency of the least common word. 4. Use binary search
comment by yorwba · 2026-04-17 · in 中文 Literacy Speedrun II: Character Cyclotron -
…be displayed as a single word, since chengyu are typically a single entry in dictionaries. [0] https://github.com/fxsjy/jieba
comment by routerl · 2025-02-04 · in Show HN: Mandarin Word Segmenter with Translation -
…which uses conditional random fields, while the most popular Chinese tokenizer is probably Jieba https://github.com/fxsjy/jieba which uses a combination of word frequencies and hidden markov models. For German, the task is much easier,…
comment by yorwba · 2020-08-14 · in How to tokenize Japanese in Python -
The segmentation is based on a library called jieba. It seems to work pretty well from my experience. https://github.com/fxsjy/jieba I do notice that some generated words are bogus (compounds of other words or redundant in some way). I…
comment by KerrickStaley · 2018-09-05 · in “Prestudy”: Learning Chinese Through Reading