Hacker GitHub

← All repos

markfasheh/duperemove

View on GitHub ↗ ★ 1,015 C

Tools for deduping file systems

9 mentions on Hacker News by 9 people: 1 story, 8 comments · first 2015-05-24 · latest 2025-02-25

Mentions per year 9 mentions · busiest year 2023 (4)
20082026
AllStoriesComments NewestTopOldest
  1. …You need a "driver" program to identify duplicates but there are a handful out there. I've used https://github.com/markfasheh/duperemove and was very pleased with how it worked.
    comment by kevincox · 2025-02-25 · in Hyperspace
  2. - https://github.com/markfasheh/duperemove - https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/ - https://github.com/adrianlopezroche/fdupes - https://github.com/pauldreik/rdfind
    comment by nikisweeting · 2024-10-31 · in OpenZFS deduplication is good now and you shouldn't use it
  3. …write-heavy disks, but it's an option. Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for…
    comment by jeroenhd · 2023-11-05 · in Fdupes: Identify or delete duplicate files
  4. …require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove
    comment by ptman · 2023-10-31 · in Bcachefs Merged into the Linux 6.7 Kernel
  5. …man-pages/man2/ioctl_fideduperange.2.... https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://github.com/markfasheh/duperemove
    comment by the8472 · 2023-07-04 · in ZFS 2.2.0 (RC): Block Cloning merged
  6. Looks like https://github.com/markfasheh/duperemove
    comment by JackSlateur · 2023-03-26 · in Craziest thing I ever used SQLite for: partial file deduplication (2022)
  7. …duplicate blocks, instead it simply implements ioctls for what is essentially extent sharing. [1] https://github.com/markfasheh/duperemove
    comment by newnewpdro · 2018-05-29 · in About ZFS Performance
  8. …btrfs from the past two years or so. [2] Much like ext*'s reserved-for-the-superuser space. [3] https://github.com/markfasheh/duperemove
    comment by simoncion · 2015-05-24 · in Ext4: fix data corruption caused by unwritten and delayed extents