hpc/libcircle
View on GitHub ↗ ★ 101 C
An API to provide an efficient distributed queue on a cluster. Libcircle is currently used in production to quickly traverse and perform operations on a file tree which contains several hundred-million file nodes.
5 mentions on Hacker News by 2 people: 2 stories, 3 comments · first 2011-09-10 · latest 2026-07-26
20082026
-
Do you know if fclones implements anything similar to https://github.com/hpc/libcircle ? I also wonder if nftw(3) would be faster than readdir(3).
comment by ranger_danger · 2026-07-26 · in DskDitto: Ultra-fast, parallel duplicate-file detector -
…be available at http://fileutils.io/ relatively soon. We also had another tool written on top of https://github.com/hpc/libcircle that would gather metadata on a few hundred-million files in a few hours (we had to limit the speed so it…
comment by fintler · 2014-09-12 · in My experience with using cp to copy 432 million files (39 TB) -
▲ 1
Libcircle supports reduction nowstory by fintler · 2014-02-19 · 0 comments
-
…for single-mount petascale filesystem treewalking that extends Dijkstra's original design: https://github.com/hpc/libcircle We have a detailed paper on it coming out at SC12.
comment by fintler · 2012-09-25 · in Google Spanner's Most Surprising Revelation: NoSQL is Out and NewSQL is In -
▲ 1
Distributed Programming With Dijkstra's Token Based Self-stabilizationstory by fintler · 2011-09-10 · 0 comments