NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

6 mentions on Hacker News by 2 people: 5 stories, 1 comment · first 2024-07-05 · latest 2024-11-12

Mentions per year 6 mentions · busiest year 2024 (6)

20082026

All Stories Comments Newest Top Oldest

…algorithm described, and I figure this community might be interested. Here's the repo: https://github.com/NVIDIA/NeMo-Curator/ Some documentation on the fuzzy dedup scripts: https://docs.nvidia.com/nemo-framework/user-guide/latest/dat...…
comment by ryantwolf · 2024-07-05 · in Finding near-duplicates with Jaccard similarity and MinHash