Hacker GitHub

← All repos

chrislusf/seaweedfs

View on GitHub ↗ ★ 45 Go

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

44 mentions on Hacker News by 21 people: 9 stories, 35 comments · first 2015-04-25 · latest 2022-12-04

Mentions per year 44 mentions · busiest year 2021 (18)
20082026
AllStoriesComments NewestTopOldest
  1. A very good alternative is seaweedfs https://github.com/chrislusf/seaweedfs/ based on facebook haystack paper (efficient small files) & more.
    comment by ddorian43 · 2022-12-04 · in Garage: An open-source distributed object storage service
  2. …to local servers, and access them at local network speed, and asynchronously sync back to S3. https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot...
    comment by chrislusf · 2022-07-29 · in S3 isn't getting cheaper
  3. …Zero PUT cost, higher operational overhead. Minio: https://github.com/minio/minio SeaweedFS: https://github.com/chrislusf/seaweedfs Ceph: https://ceph.com/en/discover/technology/
    comment by 0xbadcafebee · 2022-07-28 · in S3 isn't getting cheaper
  4. Maybe they added it since you looked or did something not match your needs? https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-f...
    comment by Already__Taken · 2022-02-09 · in Garage, our self-hosted distributed object storage solution
  5. …becomes read-only [1], thus giving up partition tolerance. These are not really comparable. [1] https://github.com/chrislusf/seaweedfs/wiki/Replication
    comment by ricardobeat · 2022-02-08 · in Garage, our self-hosted distributed object storage solution
  6. …I suggest looking into SeaweedFS[0]. It's a far more lightweight and performant solution. [0]: https://github.com/chrislusf/seaweedfs/
    comment by acejam · 2022-01-25 · in Saving on egress switching from AWS to Hetzner
  7. …you should just pick the one you are most familiar with. There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup
    comment by chrislusf · 2022-01-05 · in Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put
  8. …task is to read and write lots of small files. There is a term for it, LOSF. I work on SeaweedFS, https://github.com/chrislusf/seaweedfs , which is designed to handle LOSF. And of course, no problem with large files at all.
    comment by chrislusf · 2022-01-05 · in Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put
  9. Interesting that SeaweedFS also has a similar named "Gateway to Remote Object Storage" https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... The difference is that SeaweedFS can support both read and write, with asynchronous…
    comment by chrislusf · 2021-12-02 · in Using Nginx as an Object Storage Gateway
  10. …network speed, and reliabilities. What about using SeaweedFS gateway to remote object store? https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... Basically you can think it is a cache. You operate with normal S3 APIs, and the…
    comment by chrislusf · 2021-10-07 · in Will Cloudflare R2 Win Customers from Amazon S3?
  11. …your data fast and cheap is not easy. That is why SeaweedFS added a gateway to remote object store. https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remot... , which asynchronously write local changes to the cloud. If there are…
    comment by chrislusf · 2021-09-28 · in Cloudflare R2 storage: Rapid and reliable object storage, minus the egress fees
  12. As a main contributor to an open source project ( https://github.com/chrislusf/seaweedfs ), I can confirm that this finding is so true. However, seems this research did not look into Apache projects, which basically maintain a different…
    comment by chrislusf · 2021-09-22 · in Software projects need heroes? lessons learned from 1100 projects
  13. …the volumes were calling eachother with HTTP instead of HTTPS. The HTTP request is hardcoded here: https://github.com/chrislusf/seaweedfs/blob/43fd11278ef81185... and probably in a lot of other places. I even tried setting the address of…
    comment by Fornax96 · 2021-09-01 · in Ask HN: Companies of one, what is your tech stack?
  14. …to link directly to a single storage server as a single file would also be distributed. Check out https://github.com/chrislusf/seaweedfs/ implementation of reed solomon. Small files can still be served from 1 server. It's also efficient…
    comment by ddorian43 · 2021-08-27 · in Ask HN: Companies of one, what is your tech stack?
  15. Well, the author has to be aware of it: https://github.com/chrislusf/seaweedfs/issues/2216
    comment by yencabulator · 2021-08-11 · in MinIO: A Bare Metal Drop-In for AWS S3
  16. …disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... . Both should default to fsync on, with the option to turn it off. So not a…
    comment by tobias3 · 2021-08-11 · in MinIO: A Bare Metal Drop-In for AWS S3
  17. …options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system. [0] https://github.com/chrislusf/seaweedfs
    comment by dragonsh · 2021-08-11 · in MinIO: A Bare Metal Drop-In for AWS S3
  18. Seaweedfs has made some questionable security decisions ( https://github.com/chrislusf/seaweedfs/issues/1937 ).
    comment by gunapologist99 · 2021-08-10 · in MinIO: A Bare Metal Drop-In for AWS S3
  19. …elastic scalability, checkout SeaweedFS instead, it is based on the Facebook "Haystack" paper: https://github.com/chrislusf/seaweedfs
    comment by heipei · 2021-08-10 · in MinIO: A Bare Metal Drop-In for AWS S3
  20. Thanks! I am working on SeaweedFS. https://github.com/chrislusf/seaweedfs There are also SeaweedFS CSI Driver: https://github.com/seaweedfs/seaweedfs-csi-driver
    comment by chrislusf · 2021-07-23 · in MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
  21. (I work on SeaweedFS) How about using SeaweedFS? https://github.com/chrislusf/seaweedfs With your dedicated server, the latency is consistent, No API/network cost. Extra data can be tiered to S3. SeaweedFS as a Key-Large-Value store…
    comment by chrislusf · 2021-04-28 · in Using a disk-based Redis clone to reduce AWS S3 bill
  22. …designed to store images as Facebook Haystack paper, and should be ideal for your use case. See https://github.com/chrislusf/seaweedfs And it already supports S3 API, and other HTTP, FUSE, WebDAV, Hadoop, etc. There should be many existing…
    comment by chrislusf · 2021-04-27 · in Ask HN: How would you store 10PB of data for your startup today?
  23. …metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores I did find one drawback for Cassandra as the metadata store though. One use case is…
    comment by chrislusf · 2021-02-24 · in The trouble with Cassandra as an object storage metadata database
  24. I am receiving $337 per month as of now for https://github.com/chrislusf/seaweedfs Not something to be proud of if including the time spent to evolve the project.
    comment by chrislusf · 2021-01-13 · in I received my first donation on my open-source side project
  25. …(via E.g., a relational database)? It's the problem of having too many small files. Explained here https://github.com/chrislusf/seaweedfs#introduction . > Also, what do you mean by fill/rebalance in this context? Assuming you are…
    comment by ddorian43 · 2021-01-10 · in Ask HN: Where to get cheap VPS with big storage?
  26. …them. How will you host the images ? Metadata will become a bottleneck before hdd size. Check out https://github.com/chrislusf/seaweedfs
    comment by ddorian43 · 2021-01-10 · in Ask HN: Where to get cheap VPS with big storage?
  27. …[4]: https://docs.openstack.org/swift [5]: https://github.com/dragonflyoss/Dragonfly [6]: https://github.com/chrislusf/seaweedfs#compared-to-other-fil...
    comment by hardwaresofton · 2020-11-08 · in Mayastor: Lightning Fast Storage for Kubernetes
  28. It depends on your project. I used Go since 2011 on multiple projects, two of them are http://github.com/chrislusf/seaweedfs and http://github.com/chrislusf/gleam . One is a distributed file system, the other is a distributed computation.…
    comment by chrislusf · 2020-10-25 · in Ask HN: Go programming language is over ten years old. What do you think of it?
  29. see https://github.com/chrislusf/seaweedfs/wiki/Erasure-Coding-f...
    comment by chrislusf · 2020-10-09 · in SeaweedFS – A simple and highly scalable distributed file system with S3 API
  30. …we've failed! Please raise an issue with any questions or update this file with clarifications. https://github.com/chrislusf/seaweedfs#compared-to-other-fil... However, since I never had to touch hdfs after installing it in the first…
    comment by 0x008 · 2020-10-09 · in SeaweedFS – A simple and highly scalable distributed file system with S3 API
  31. Actually there is a pull request to add redis support to list files. https://github.com/chrislusf/seaweedfs/pull/254 I just haven't have time to test it and make it work yet. If you want to, you can help to verify this.
    comment by chrislusf · 2017-01-05 · in SeaweedFS – A simple and highly scalable distributed file system
  32. …piece that does provide at least directories. Still short of being a "filesystem" IMO, though: https://github.com/chrislusf/seaweedfs/wiki/Filer
    comment by tyingq · 2017-01-04 · in SeaweedFS – A simple and highly scalable distributed file system
  33. Re protection, see: https://github.com/chrislusf/seaweedfs#rack-aware-and-data-c... (Not an author or anything, just found this project today, and it's interesting enough for me to test it out.) Edit to add: I've done some testing,…
    comment by jdcarter · 2017-01-04 · in SeaweedFS – A simple and highly scalable distributed file system
  34. …a simple config file and be done with it. SeaweedFS shines a light on how this could be improved: https://github.com/chrislusf/seaweedfs
    comment by kyledrake · 2016-10-20 · in New storage classes for Google Cloud Storage
  35. SeaweedFS Distributed File System https://github.com/chrislusf/seaweedfs Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast!
    comment by chrislusf · 2015-04-25 · in Ask HN: Brag about open-source projects you're proud of