NewbiZ/s3pd
View on GitHub ↗ ★ 9 Python
S3 parallel downloader with good performance
4 mentions on Hacker News by 1 person: 0 stories, 4 comments · first 2020-11-22 · latest 2025-06-11
20082026
-
…while spawning 20 or so threads which were all idling for IO. The same test using a Python tool ( http://github.com/NewbiZ/s3pd ) with the same amount of workers took 10s. Cranking up the worker count of the latter library until there is…
comment by Galanwe · 2025-06-11 · in S5cmd: Parallel S3 and local filesystem execution tool -
Using multiprocessing I've been able to quite easily saturate a 20GBps Ec2 NICs in python. https://github.com/NewbiZ/s3pd There is no reason why multiprocessing for IO in python would use _crazily_ more memory than in an other language,…
comment by Galanwe · 2021-04-11 · in Downloading files from S3 with multithreading and Boto3 -
That's what my small python package does ( https://github.com/NewbiZ/s3pd ). Definitely not perfect (not saturating cores because no use of event loop per process) but download is split in multiple processes and stored in shared memory.…
comment by Galanwe · 2021-04-11 · in Downloading files from S3 with multithreading and Boto3 -
…have multiple parallel blocks downloaded. I wrote a simple library that does this called s3pd ( https://github.com/NewbiZ/s3pd ). It's not perfect and is process based instead of coroutines, but that will give you an idea. For reference,…
comment by Galanwe · 2020-11-22 · in HopsFS: 100x Times Faster Than AWS S3