Hacker GitHub

← All repos

meta/zstd

View on GitHub ↗ ★ 28,151 C

Zstandard - Fast real-time compression algorithm

165 mentions on Hacker News by 112 people: 28 stories, 137 comments · first 2016-08-31 · latest 2026-09-07

Mentions per year 165 mentions · busiest year 2018 (27)
20082026
AllStoriesComments NewestTopOldest
  1. For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli , is another modern format with some good features for…
    comment by twotwotwo · 2022-10-11 · in How much faster is making a tar archive without gzip?
  2. …Using this dictionary, the compression ratio achievable on small data improves dramatically." https://github.com/facebook/zstd#the-case-for-small-data-com...
    comment by cyann · 2022-09-05 · in Zstandard – Real-time data compression algorithm
  3. zstd: https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
    comment by VHRanger · 2022-08-23 · in Libgen-text: complete Library Genesis, re-packaged as text only (1.2TB)
  4. zstd 1.3.8 and later also has an rsync-friendly mode (via the --rsyncable command-line option). See https://github.com/facebook/zstd/issues/1155
    comment by kevinoid · 2022-08-20 · in AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
  5. https://github.com/facebook/zstd/issues/1333
    comment by hdante · 2022-08-20 · in AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
  6. zstd still doesn't have a seekable format as part of the official standard (I wish it did): https://github.com/facebook/zstd/issues/395#issuecomment-535...
    comment by rwmj · 2022-07-24 · in Xz format considered inadequate for long-term archiving (2016)
  7. …if you want (very) fast compression/decompression and/or general high-ratio compression, use .zstd. https://github.com/facebook/zstd I've used this combination professionally to great effect. You're welcome :)
    comment by metadat · 2022-07-24 · in Xz format considered inadequate for long-term archiving (2016)
  8. I’m sure autoconf will insert this if you explicitly tell it to [1]. 1: https://github.com/facebook/zstd/blob/b7b7edb3a3017ac8e16d7e...
    comment by nemetroid · 2022-06-19 · in The Bizarre Case of Zstd's Slow Performance on Arch Linux
  9. …only since C11. For older standards, it uses clock(); but that returns CPU time, which is wrong. https://github.com/facebook/zstd/pull/3165
    comment by jwilk · 2022-06-19 · in The Bizarre Case of Zstd's Slow Performance on Arch Linux
  10. …shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd But most people I work with will pick zstd every time even in cases where decompression speed matters the…
    comment by CoolGuySteve · 2022-05-25 · in Lizard – efficient compression with fast decompression
  11. …" (...) includes dictionary builder, benchmark, and supports decompression of legacy zstd formats " https://github.com/facebook/zstd/tree/dev/programs The project also provides another build target, zstd-small, which is " CLI optimized for…
    comment by arinlen · 2022-05-19 · in Zstandard Worked Example
  12. https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory. $ size harness text data bss dec hex filename 24615 704…
    comment by nigeltao · 2022-05-19 · in Zstandard Worked Example
  13. The official cli windows binaries are here: https://github.com/facebook/zstd/releases/ Afaik, they don't need any dependencies.
    comment by wolf550e · 2022-05-19 · in Zstandard Worked Example
  14. …this question: I can haz zstd seeks plz? [1] https://github.com/martinellimarco/libzstd-seek [2] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... [3] https://github.com/zraorg/ZRA [4] https://github.com/facebook/zstd/issues/3…
    comment by mananaysiempre · 2022-05-19 · in Zstandard Worked Example
  15. …in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful. [0] https://github.com/facebook/zstd/issues/3133
    comment by terrelln · 2022-05-19 · in Zstandard Worked Example
  16. …#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX) [3] #define ZSTD_BLOCKSIZELOG_MAX 17 [3] [0] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [1] https://github.com/facebook/zstd/blob/550410d05d7c7815b1ff41... [2]…
    comment by aidanhs · 2022-05-19 · in Zstandard Worked Example
  17. …legal to omit the checksum, see the spec). That would get the total overhead down to 5 bytes. [1]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...
    comment by pcwalton · 2022-05-18 · in Zstandard Worked Example
  18. Really wish this was part of official zstd ( https://github.com/facebook/zstd/issues/395#issuecomment-535... ) and not a contrib / separate tool.
    comment by rwmj · 2022-04-23 · in Casync – A Content-Addressable Data Synchronization Tool
  19. …w/ negligible CPU usage increase. [0] https://github.com/SaveTheRbtz/zstd-seekable-format-go [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... [2] e.g FastCDC https://www.usenix.org/system/files/conference/atc16/atc16-p...…
    comment by SaveTheRbtz · 2022-04-23 · in Casync – A Content-Addressable Data Synchronization Tool
  20. And the benchmark has been updated : https://github.com/facebook/zstd#benchmarks - lz4 1.9.3 ; zstd 1.5.1 -1 ; zlib 1.2.11 -1 ; ...
    comment by pella · 2021-12-21 · in Zstandard v1.5.1
  21. I am amazed at the variety of repos shared - by you, as well as others. Thanks! Repo links: https://github.com/facebook/zstd https://github.com/google/brotli https://github.com/ClickHouse/ClickHouse https://github.com/evanw/esbuild…
    comment by pdevr · 2021-08-24 · in Ask HN: What are the best publicly available FAMANG code repos?
  22. You can compile only the decompressor of the official ZSTD library to wasm, it is ony 28 kB: https://github.com/facebook/zstd/files/6602821/zstd.zip
    comment by IvanK_net · 2021-07-03 · in Compressing JSON: Gzip vs. Zstd
  23. …of the reference corpus. [1] 5. It has (more than one) python binding available. [2][3] [0] https://github.com/facebook/zstd [1] https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L40 [2] https://pypi.org/project/zstandard [3]…
    comment by felixhandte · 2021-06-09 · in Text Classification by Data Compression
  24. …the most trivial of projects. Isn't BSD widely accepted to have an implicit patent license? e.g.: https://github.com/facebook/zstd/issues/335
    comment by rockdoe · 2021-04-13 · in Linus Torvalds says GPL v3 violates everything that GPLv2 stood for (2014)
  25. …around for ages, implementations can still improve, especially as the hardware we use changes. [0] https://github.com/facebook/zstd/pull/2271
    comment by terrelln · 2021-02-22 · in What Is Huffman Coding?
  26. …more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data…
    comment by jakozaur · 2021-01-27 · in LZ4 – Extremely fast compression
  27. another contender is zstd: https://github.com/facebook/zstd . It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to…
    comment by roncohen · 2021-01-27 · in LZ4 – Extremely fast compression
  28. …if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time. [0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...
    comment by felixhandte · 2021-01-03 · in ECC matters
  29. Zstandard is pretty nice: https://github.com/facebook/zstd
    comment by abrowne · 2020-12-24 · in Facebook Declared War on Apple, and It Just May Signal the End of Facebook
  30. …other delta engines. There is also a wiki page detailing how to use zstd as a delta engine [1]. [0] https://github.com/facebook/zstd/releases/tag/v1.4.5 [1] https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...
    comment by terrelln · 2020-12-17 · in Zstandard v1.4.7
  31. …native performance! A testament to the wasm engine, I guess. There's discussion on this topic here: https://github.com/facebook/zstd/issues/1355
    comment by felixhandte · 2020-12-17 · in Zstandard v1.4.7
  32. https://github.com/facebook/zstd/issues/395#issuecomment-535... If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really…
    comment by rwmj · 2020-12-17 · in Zstandard v1.4.7
  33. There's a web assembly version in the repository : https://github.com/facebook/zstd/tree/dev/contrib/single_fil... Could be worth checking
    comment by twitch_checksum · 2020-10-10 · in Facebook/Zstandard – Fast real-time compression algorithm
  34. …for seekable (and thus parallel) decompression, but it isn't included in distributed binaries. See: https://github.com/facebook/zstd/issues/395
    comment by goodside · 2020-07-12 · in Parallel Gzip – Pigz
  35. …code in the `contrib` dir for parallel (and, more importantly, seekable) decompression: https://github.com/facebook/zstd/issues/395
    comment by goodside · 2020-07-12 · in Parallel Gzip – Pigz
  36. …W.r.t. to ZSTD: The usual thing is to provide a zlib compatible API, which ZSTD does ( https://github.com/facebook/zstd/tree/dev/zlibWrapper ). But the ZSTD zlib compatibility layer causes lower performance, so it is better to use it…
    comment by tobias3 · 2020-05-17 · in Introduce ZSTD compression to ZFS
  37. …by the creator of Node.js) https://deno.land/ - zstd, a relatively new compression algorithm https://github.com/facebook/zstd https://quixdb.github.io/squash-benchmark/ - BLAKE2b, which is the new hashing function you should be using. It's…
    comment by shpx · 2020-01-05 · in Ask HN: What Technologies to Learn in 2020?
  38. …ratio and compression speed, but its decompression speed is significantly lower and closer to zlib. https://github.com/facebook/zstd#benchmarks AFAIK (I haven't looked much into it since 2018) it's not widely supported by CDNs, but at…
    comment by dictum · 2020-01-05 · in Now using Zstandard instead of xz for package compression
  39. I wish zstd supported seeking and partial decompression ( https://github.com/facebook/zstd/issues/395#issuecomment-535... ). We could then use it for hosting disk images as it would be a lot faster than xz which we currently use.
    comment by rwmj · 2020-01-05 · in Now using Zstandard instead of xz for package compression
  40. …to what you're going to compress; I would guess they contain something resembling frequency tables. https://github.com/facebook/zstd has some description but doesn't explain precisely what the dictionary contains.
    comment by waterhouse · 2019-08-11 · in Hutter Prize: Compress a 100MB file to less than the current record of 16 MB
  41. …its use in production. And then there are two re-implementations in Java [1] and Go [2]. [0] https://github.com/facebook/zstd/tree/dev/doc/educational_de... [1] https://github.com/airlift/aircompressor/tree/master/src/mai... [2]…
    comment by felixhandte · 2019-07-20 · in Zstandard v1.4.1
  42. …of input?). ---- It's also worth pointing out that zstd also has plenty of options to fiddle with: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
    comment by vthriller · 2019-04-17 · in Zstandard v1.4.0
  43. …by a guy at Facebook and seems to be owned by Facebook as well, since it's hosted on their github: https://github.com/facebook/zstd On the other hand, according to https://facebook.github.io/zstd it seems that zstd is part of the Linux…
    comment by Xolvix · 2019-03-25 · in Arch Linux propose changing compression method from xz to zstd
  44. …plots that I work with to evaluate zstd performance, e.g. [2]. [1] https://imgur.com/gU2Gdf6 [2] https://github.com/facebook/zstd/pull/1317#issuecomment-4260...
    comment by felixhandte · 2018-12-20 · in Improving compression at scale with Zstandard
  45. Ha, that just came up: https://github.com/facebook/zstd/issues/1087
    comment by felixhandte · 2018-12-19 · in Improving compression at scale with Zstandard
  46. …will be a benchmark you run yourself with your own data. And hopefully you like what you see! [1] https://github.com/facebook/zstd/releases/tag/v1.3.6
    comment by felixhandte · 2018-12-19 · in Improving compression at scale with Zstandard
  47. …* Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper
    comment by terrelln · 2018-12-19 · in Improving compression at scale with Zstandard
  48. There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper
    comment by Cyan4973 · 2018-12-19 · in Improving compression at scale with Zstandard
  49. This may help? https://github.com/facebook/zstd/issues/1155
    comment by aberoham · 2018-11-26 · in Zstandard – Fast real-time compression algorithm
  50. …--long=31 -T4 -10 It should be faster than lrzip + zstd and provide about the same results. [0] https://github.com/facebook/zstd/releases/tag/v1.3.2
    comment by terrelln · 2018-11-26 · in Zstandard – Fast real-time compression algorithm