Hacker GitHub

← All repos

rlmcpherson/s3gof3r

View on GitHub ↗ ★ 1,142 Go

Fast, concurrent, streaming access to Amazon S3, including gof3r, a CLI. http://godoc.org/github.com/rlmcpherson/s3gof3r

9 mentions on Hacker News by 3 people: 1 story, 8 comments · first 2014-02-06 · latest 2016-04-19

Mentions per year 9 mentions · busiest year 2014 (4)
20082026
AllStoriesComments NewestTopOldest
  1. try https://github.com/rlmcpherson/s3gof3r you can often get wire speed
    comment by hrez · 2016-04-19 · in Amazon S3 Transfer Acceleration
  2. …done over a year ago so bandwidth limitations on EC2 may have changed since. This testing was with https://github.com/rlmcpherson/s3gof3r
    comment by rmcpherson · 2016-01-06 · in Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
  3. Excellent https://github.com/rlmcpherson/s3gof3r is my tool of choice for "fast, parallelized, pipelined streaming access to Amazon S3." If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.
    comment by hrez · 2016-01-05 · in Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage
  4. To be fair, here is a tool that has features no other tool I know: https://github.com/rlmcpherson/s3gof3r It can saturate your bandwidth and you can pipe to/from it. It's great to deal with hundreds GB's archive on the fly. "Examples: $…
    comment by hrez · 2015-09-20 · in S3concurrent, yet another S3 syncing project
  5. I haven't used this library, but this appears to be what you want. https://github.com/rlmcpherson/s3gof3r Nice features include: > Retry Everything: All http requests and every part is retried on both uploads and downloads. > Configurable…
    comment by placeybordeaux · 2015-07-07 · in Handling 1M Requests per Minute with Go
  6. ▲ 1
    S3gof3r: Streaming, parallel S3 uploads and downloads at 1 Gbps+
    story by rmcpherson · 2014-02-18 · 0 comments
  7. I missed your question about build instructions, sorry. I'll add them to the readme, but it's just "go get github.com/rlmcpherson/s3gof3r/gof3r" for the CLI once you have installed go. Usage instructions are in the readme already.
    comment by rmcpherson · 2014-02-08 · in How to Save 90% on your S3 Bill
  8. …so that may be what you are referring to. These have a minimum size set by amazon of 5 MB. With the https://github.com/rlmcpherson/s3gof3r the memory overhead of a streaming upload is approximately part size * concurrent uploads (plus a…
    comment by rmcpherson · 2014-02-06 · in How to Save 90% on your S3 Bill
  9. Because of the lack of fast and reliable s3 clients out there, I built https://github.com/rlmcpherson/s3gof3r , that can do over 1 Gbps with ease, both multipart uploads and parallelized downloads. The killer feature, though, is streaming…
    comment by rmcpherson · 2014-02-06 · in How to Save 90% on your S3 Bill