Hacker GitHub

← All repos

NVIDIA-NeMo/Curator

View on GitHub ↗ ★ 1,856 Python

Scalable data pre processing and curation toolkit for LLMs

6 mentions on Hacker News by 2 people: 5 stories, 1 comment · first 2024-07-05 · latest 2024-11-12

Mentions per year 6 mentions · busiest year 2024 (6)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 2
    Fast and scalable dataset preparation and curation tool from Nvidia
    story by shcheklein · 2024-11-12 · 0 comments
  2. ▲ 12
    Fast and scalable dataset preparation and curation tool from Nvidia
    story by shcheklein · 2024-10-23 · 0 comments
  3. ▲ 1
    Scalable data pre processing and curation toolkit for AI
    story by shcheklein · 2024-09-17 · 0 comments
  4. ▲ 1
    Scalable data pre processing and curation toolkit for LLMs
    story by shcheklein · 2024-08-21 · 0 comments
  5. ▲ 2
    Scalable Toolkit for Data Curation
    story by shcheklein · 2024-08-08 · 0 comments
  6. …algorithm described, and I figure this community might be interested. Here's the repo: https://github.com/NVIDIA/NeMo-Curator/ Some documentation on the fuzzy dedup scripts: https://docs.nvidia.com/nemo-framework/user-guide/latest/dat...…
    comment by ryantwolf · 2024-07-05 · in Finding near-duplicates with Jaccard similarity and MinHash