visual-layer/fastdup
View on GitHub ↗ ★ 1,915 Python
fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.
4 mentions on Hacker News by 2 people: 2 stories, 2 comments · first 2022-12-13 · latest 2023-05-02
20082026
-
▲ 1
Visualize your dataset using DINOv2 embeddingstory by dnth · 2023-05-02 · 0 comments
-
▲ 2
Find image duplicates and outliers – A free, scalable, efficient toolstory by dnth · 2023-03-21 · 1 comment
-
…discrepancies. ImageNet21K > 1.2M duplicate images. > 104K train/val leak. fastdup GitHub repo - https://github.com/visual-layer/fastdup
comment by dnth · 2023-03-07 · in Dedup-ing LAION (60M duplicates) and ImageNet (1.2M duplicates) with fastdup -
…of the data quality tool for computer vision, fastdup, continue to improve on their free release https://github.com/visual-layer/fastdup Here's a short video of some recent results for LAION 400M https://www.youtube.com/watch?v=dlRCm29Upu4
comment by datascientist · 2022-12-13 · in Laion-5B: A new era of open large-scale multi-modal datasets