hpc/mpifileutils
View on GitHub ↗ ★ 203 C
File utilities designed for scalability and performance.
4 mentions on Hacker News by 2 people: 1 story, 3 comments · first 2021-08-15 · latest 2022-10-18
20082026
-
…You can run it across multiple machines and it'll automatically balance the workload across them. https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1
comment by fintler · 2022-10-18 · in pigz: A parallel implementation of gzip for multi-core machines -
If you want something that scales horizontally (mostly), dcmp from https://github.com/hpc/mpifileutils is an option. It can chunk up files and do the comparison in parallel on multiple servers.
comment by fintler · 2021-08-30 · in Go Find Duplicates: A fast and simple tool to find duplicate files -
https://github.com/hpc/mpifileutils handles this pretty well -- with SYS_getdents64. It has a few other tricks in there in addition to this one.
comment by fintler · 2021-08-15 · in You can list a directory containing 8M files, but not with ls