rspeer/wordfreq
View on GitHub ↗ ★ 1,766 Python
Access a database of word frequencies, in various natural languages.
7 mentions on Hacker News by 4 people: 1 story, 6 comments · first 2016-12-17 · latest 2025-06-11
20082026
-
…to find. What with archive.org, gutenberg etc. But recent uncontaminated data is hard to find. https://github.com/rspeer/wordfreq/blob/master/SUNSET.md
comment by fer · 2025-06-11 · in Low-background Steel: content without AI contamination -
…[1]. The only exception from your list would be OCR where a human proofreads the output. [1] https://github.com/rspeer/wordfreq/blob/master/SUNSET.md
comment by a57721 · 2025-06-11 · in Low-background Steel: content without AI contamination -
…drop sometimes) on realistic data. It's much faster than Python's LRU decorator, for example. [1] https://github.com/LuminosoInsight/wordfreq/blob/master/word...
comment by rspeer · 2017-04-19 · in Cache eviction: when are randomized algorithms better than LRU? (2014) -
…and generally to be able to measure differences in register between child and adult language. [1] https://github.com/LuminosoInsight/wordfreq
comment by rspeer · 2017-02-24 · in Wordbank: An open database of children's vocabulary development -
…multiple surrogate code points, which are not characters), if that's what you're asking about. [0] https://github.com/LuminosoInsight/wordfreq/blob/master/word... [UAX 29] http://unicode.org/reports/tr29/ (fixed link) [PEP 393]…
comment by rspeer · 2016-12-17 · in Natural Language Corpus Data: Beautiful Data -
…[2] http://drhagen.com/blog/the-missing-11th-of-the-month/ [3] https://github.com/LuminosoInsight/wordfreq [4] http://opus.lingfil.uu.se/OpenSubtitles2016.php
comment by rspeer · 2016-12-17 · in Natural Language Corpus Data: Beautiful Data