huggingface/datasets
View on GitHub ↗ ★ 22,104 Python
🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools
11 mentions on Hacker News by 8 people: 6 stories, 5 comments · first 2020-05-15 · latest 2026-07-23
20082026
-
…pickle hole looks possible. Their datasets library uses the pandas.read_pickle() method here: https://github.com/huggingface/datasets/blob/d21c5816d5d1961...
comment by simonw · 2026-07-23 · in OpenAI’s accidental attack against Hugging Face is science fiction that happened -
This is my (current) favorite list comprehension: https://github.com/huggingface/datasets/blob/871eabc7b23c27d... Someone was feeling awfully clever that day. (Not that I'm not occasionally guilty myself.)
comment by dpkirchner · 2024-08-13 · in Go is my hammer, and everything is a nail -
https://github.com/huggingface/datasets https://github.com/huggingface/transformers
comment by swyx · 2023-02-12 · in How to train large models on many GPUs? (2021) -
"HuggingFace datasets" is an open source Python package: https://github.com/huggingface/datasets/ And they also have ready-to-use scripts for A LOT of the usual datasets: https://huggingface.co/datasets including LAION 400M and LAION 2B:…
comment by fxtentacle · 2022-11-15 · in Launch HN: Activeloop (YC S18) – Data lake for deep learning -
…{"text": "contents_of_source_file_2"} ... And then pass that my_code.json as the dataset name. [1] https://github.com/huggingface/datasets
comment by moyix · 2022-08-03 · in FauxPilot – an attempt to build a locally hosted version of GitHub Copilot