Hacker GitHub

← All repos

meta/zstd

View on GitHub ↗ ★ 28,151 C

Zstandard - Fast real-time compression algorithm

165 mentions on Hacker News by 112 people: 28 stories, 137 comments · first 2016-08-31 · latest 2026-09-07

Mentions per year 165 mentions · busiest year 2018 (27)
20082026
AllStoriesComments NewestTopOldest
  1. They have a nice table on that page: https://github.com/facebook/zstd#benchmarks Looking at that table, I think LZ4 is a winner. The compression ratio is not too far, compression speed is slightly faster, decompression speed is…
    comment by Const-me · 2022-12-20 · in Zstandard – Fast real-time compression algorithm
  2. 2.97. https://github.com/facebook/zstd/issues/1087#issuecomment-44...
    comment by CharlesW · 2022-12-20 · in Zstandard – Fast real-time compression algorithm
  3. …no one uses the 4-clause BSD (the problematic one). The one Zstd uses has three clauses: https://github.com/facebook/zstd/blob/dev/LICENSE https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_...
    comment by loeg · 2022-12-20 · in Zstandard – Fast real-time compression algorithm
  4. ▲ 227
    Zstandard – Fast real-time compression algorithm
    story by tosh · 2022-12-19 · 93 comments
  5. zstd still doesn't support seekable compressed disk images, the main advantage that xz has: https://github.com/facebook/zstd/issues/395#issuecomment-535...
    comment by rwmj · 2022-12-04 · in Creating a modifiable gzipped disk image
  6. Fortunately, zstd seems to be in active development: https://github.com/facebook/zstd As far as I can tell, most of zstd's development is still by Facebook employees, though not all of it. I tend to think zstd has enough traction that…
    comment by loudmax · 2022-11-15 · in Sapling: A new source control system with Git-compatible client
  7. …https://libguestfs.org/nbdkit-xz-filter.1.html Unfortunately zstd still does not: https://github.com/facebook/zstd/issues/395#issuecomment-535...
    comment by rwmj · 2022-10-11 · in How much faster is making a tar archive without gzip?
  8. For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli , is another modern format with some good features for…
    comment by twotwotwo · 2022-10-11 · in How much faster is making a tar archive without gzip?
  9. …Using this dictionary, the compression ratio achievable on small data improves dramatically." https://github.com/facebook/zstd#the-case-for-small-data-com...
    comment by cyann · 2022-09-05 · in Zstandard – Real-time data compression algorithm
  10. zstd: https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
    comment by VHRanger · 2022-08-23 · in Libgen-text: complete Library Genesis, re-packaged as text only (1.2TB)
  11. ▲ 2
    Zstandard – Fast real-time compression algorithm
    story by doener · 2022-08-22 · 0 comments
  12. zstd 1.3.8 and later also has an rsync-friendly mode (via the --rsyncable command-line option). See https://github.com/facebook/zstd/issues/1155
    comment by kevinoid · 2022-08-20 · in AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
  13. https://github.com/facebook/zstd/issues/1333
    comment by hdante · 2022-08-20 · in AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
  14. zstd still doesn't have a seekable format as part of the official standard (I wish it did): https://github.com/facebook/zstd/issues/395#issuecomment-535...
    comment by rwmj · 2022-07-24 · in Xz format considered inadequate for long-term archiving (2016)
  15. …if you want (very) fast compression/decompression and/or general high-ratio compression, use .zstd. https://github.com/facebook/zstd I've used this combination professionally to great effect. You're welcome :)
    comment by metadat · 2022-07-24 · in Xz format considered inadequate for long-term archiving (2016)
  16. I’m sure autoconf will insert this if you explicitly tell it to [1]. 1: https://github.com/facebook/zstd/blob/b7b7edb3a3017ac8e16d7e...
    comment by nemetroid · 2022-06-19 · in The Bizarre Case of Zstd's Slow Performance on Arch Linux
  17. …only since C11. For older standards, it uses clock(); but that returns CPU time, which is wrong. https://github.com/facebook/zstd/pull/3165
    comment by jwilk · 2022-06-19 · in The Bizarre Case of Zstd's Slow Performance on Arch Linux
  18. …shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd But most people I work with will pick zstd every time even in cases where decompression speed matters the…
    comment by CoolGuySteve · 2022-05-25 · in Lizard – efficient compression with fast decompression
  19. …" (...) includes dictionary builder, benchmark, and supports decompression of legacy zstd formats " https://github.com/facebook/zstd/tree/dev/programs The project also provides another build target, zstd-small, which is " CLI optimized for…
    comment by arinlen · 2022-05-19 · in Zstandard Worked Example
  20. https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory. $ size harness text data bss dec hex filename 24615 704…
    comment by nigeltao · 2022-05-19 · in Zstandard Worked Example
  21. The official cli windows binaries are here: https://github.com/facebook/zstd/releases/ Afaik, they don't need any dependencies.
    comment by wolf550e · 2022-05-19 · in Zstandard Worked Example
  22. …this question: I can haz zstd seeks plz? [1] https://github.com/martinellimarco/libzstd-seek [2] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... [3] https://github.com/zraorg/ZRA [4] https://github.com/facebook/zstd/issues/3…
    comment by mananaysiempre · 2022-05-19 · in Zstandard Worked Example
  23. …in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful. [0] https://github.com/facebook/zstd/issues/3133
    comment by terrelln · 2022-05-19 · in Zstandard Worked Example
  24. …#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX) [3] #define ZSTD_BLOCKSIZELOG_MAX 17 [3] [0] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [1] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [2]…
    comment by aidanhs · 2022-05-19 · in Zstandard Worked Example
  25. …legal to omit the checksum, see the spec). That would get the total overhead down to 5 bytes. [1]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...
    comment by pcwalton · 2022-05-18 · in Zstandard Worked Example
  26. Really wish this was part of official zstd ( https://github.com/facebook/zstd/issues/395#issuecomment-535... ) and not a contrib / separate tool.
    comment by rwmj · 2022-04-23 · in Casync – A Content-Addressable Data Synchronization Tool
  27. …w/ negligible CPU usage increase. [0] https://github.com/SaveTheRbtz/zstd-seekable-format-go [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... [2] e.g FastCDC https://www.usenix.org/system/files/conference/atc16/atc16-p...…
    comment by SaveTheRbtz · 2022-04-23 · in Casync – A Content-Addressable Data Synchronization Tool
  28. ▲ 5
    Zstandard v1.5.1
    story by felixhandte · 2021-12-21 · 0 comments
  29. And the benchmark has been updated : https://github.com/facebook/zstd#benchmarks - lz4 1.9.3 ; zstd 1.5.1 -1 ; zlib 1.2.11 -1 ; ...
    comment by pella · 2021-12-21 · in Zstandard v1.5.1
  30. ▲ 6
    Zstandard v1.5.1
    story by ascom · 2021-12-21 · 1 comment
  31. I am amazed at the variety of repos shared - by you, as well as others. Thanks! Repo links: https://github.com/facebook/zstd https://github.com/google/brotli https://github.com/ClickHouse/ClickHouse https://github.com/evanw/esbuild…
    comment by pdevr · 2021-08-24 · in Ask HN: What are the best publicly available FAMANG code repos?
  32. You can compile only the decompressor of the official ZSTD library to wasm, it is ony 28 kB: https://github.com/facebook/zstd/files/6602821/zstd.zip
    comment by IvanK_net · 2021-07-03 · in Compressing JSON: Gzip vs. Zstd
  33. …of the reference corpus. [1] 5. It has (more than one) python binding available. [2][3] [0] https://github.com/facebook/zstd [1] https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L40 [2] https://pypi.org/project/zstandard [3]…
    comment by felixhandte · 2021-06-09 · in Text Classification by Data Compression
  34. ▲ 150
    Zstandard v1.5.0
    story by ascom · 2021-05-14 · 36 comments
  35. …the most trivial of projects. Isn't BSD widely accepted to have an implicit patent license? e.g.: https://github.com/facebook/zstd/issues/335
    comment by rockdoe · 2021-04-13 · in Linus Torvalds says GPL v3 violates everything that GPLv2 stood for (2014)
  36. …around for ages, implementations can still improve, especially as the hardware we use changes. [0] https://github.com/facebook/zstd/pull/2271
    comment by terrelln · 2021-02-22 · in What Is Huffman Coding?
  37. …more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data…
    comment by jakozaur · 2021-01-27 · in LZ4 – Extremely fast compression
  38. another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to…
    comment by roncohen · 2021-01-27 · in LZ4 – Extremely fast compression
  39. …if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time. [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...
    comment by felixhandte · 2021-01-03 · in ECC matters
  40. Zstandard is pretty nice: https://github.com/facebook/zstd
    comment by abrowne · 2020-12-24 · in Facebook Declared War on Apple, and It Just May Signal the End of Facebook
  41. …other delta engines. There is also a wiki page detailing how to use zstd as a delta engine [1]. [0] https://github.com/facebook/zstd/releases/tag/v1.4.5 [1] https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...
    comment by terrelln · 2020-12-17 · in Zstandard v1.4.7
  42. …native performance! A testament to the wasm engine, I guess. There's discussion on this topic here: https://github.com/facebook/zstd/issues/1355
    comment by felixhandte · 2020-12-17 · in Zstandard v1.4.7
  43. https://github.com/facebook/zstd/issues/395#issuecomment-535... If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really…
    comment by rwmj · 2020-12-17 · in Zstandard v1.4.7
  44. ▲ 211
    Zstandard v1.4.7
    story by varbhat · 2020-12-17 · 73 comments
  45. There's a web assembly version in the repository : https://github.com/facebook/zstd/tree/dev/contrib/single_fil... Could be worth checking
    comment by twitch_checksum · 2020-10-10 · in Facebook/Zstandard – Fast real-time compression algorithm
  46. ▲ 56
    Facebook/Zstandard – Fast real-time compression algorithm
    story by animationwill · 2020-10-08 · 13 comments
  47. …for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395
    comment by goodside · 2020-07-12 · in Parallel Gzip – Pigz
  48. …code in the `contrib` dir for parallel (and, more importantly, seekable) decompression: https://github.com/facebook/zstd/issues/395
    comment by goodside · 2020-07-12 · in Parallel Gzip – Pigz
  49. ▲ 3
    Zstandard v1.4.5 Released
    story by reddotX · 2020-05-22 · 0 comments
  50. …W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it…
    comment by tobias3 · 2020-05-17 · in Introduce ZSTD compression to ZFS