rlmcpherson/s3gof3r
View on GitHub ↗ ★ 1,142 Go
Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r
9 mentions on Hacker News by 3 people: 1 story, 8 comments · first 2014-02-06 · latest 2016-04-19
20082026
-
try https://github.com/rlmcpherson/s3gof3r you can often get wire speed
comment by hrez · 2016-04-19 · in Amazon S3 Transfer Acceleration -
…done over a year ago so bandwidth limitations on EC2 may have changed since. This testing was with https://github.com/rlmcpherson/s3gof3r
comment by rmcpherson · 2016-01-06 · in Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage -
Excellent https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3." If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.
comment by hrez · 2016-01-05 · in Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage -
To be fair, here is a tool that has features no other tool I know: https://github.com/rlmcpherson/s3gof3r It can saturate your bandwidth and you can pipe to/from it. It's great to deal with hundreds GB's archive on the fly. "Examples: $…
comment by hrez · 2015-09-20 · in S3concurrent, yet another S3 syncing project -
I haven't used this library, but this appears to be what you want. https://github.com/rlmcpherson/s3gof3r Nice features include: > Retry Everything: All http requests and every part is retried on both uploads and downloads. > Configurable…
comment by placeybordeaux · 2015-07-07 · in Handling 1M Requests per Minute with Go -
I missed your question about build instructions, sorry. I'll add them to the readme, but it's just "go get github.com/rlmcpherson/s3gof3r/gof3r" for the CLI once you have installed go. Usage instructions are in the readme already.
comment by rmcpherson · 2014-02-08 · in How to Save 90% on your S3 Bill -
…so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a…
comment by rmcpherson · 2014-02-06 · in How to Save 90% on your S3 Bill -
Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r , that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming…
comment by rmcpherson · 2014-02-06 · in How to Save 90% on your S3 Bill