Hacker GitHub

← All repos

meta/zstd

View on GitHub ↗ ★ 28,151 C

Zstandard - Fast real-time compression algorithm

165 mentions on Hacker News by 112 people: 28 stories, 137 comments · first 2016-08-31 · latest 2026-09-07

Mentions per year 165 mentions · busiest year 2018 (27)
20082026
AllStoriesComments NewestTopOldest
  1. > 8MB I believe for high levels Yep, i found it in the source here: - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... - https://github.com/facebook/zstd/blob/d9c0c7e2cf8a8bf9fb98d3... Also, zstd docs say: > Note: If…
    comment by nvme0n1p1 · 2026-09-07 · in bzip3
  2. …1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
    comment by NostraDavid · 2026-09-03 · in We could save petabytes of cache storage with Zstandard and Pingora
  3. …data. Zstd has a seekable format for frames, similar to pigz --independent works. [1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
    comment by gopalv · 2026-09-02 · in We could save petabytes of cache storage with Zstandard and Pingora
  4. …about random access at that link. There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...
    comment by ncruces · 2026-09-02 · in We could save petabytes of cache storage with Zstandard and Pingora
  5. …every third-party port in Rust, Go, Java, JS." and offered a working link [1] to the former. [1] https://github.com/facebook/zstd/blob/dev/doc/educational_decoder/zstd_decompress.c
    comment by lowbloodsugar · 2026-08-11 · in What's the best programming language for coding agents?
  6. …than the established tools like gzip (1992), bzip2 (1996), LZMA as used by xz utils (1999) Edit: https://github.com/facebook/zstd/blob/4856a00164c1d7b947bd38... initial commit indeed states it's meant to have good ratios and good…
    comment by lucb1e · 2026-03-15 · in An ode to bzip
  7. …It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting. https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...
    comment by laurencerowe · 2026-03-14 · in An ode to bzip
  8. …can recognize and separate that framing to recover a contiguous view of the underlying data. [0] https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zs... (I am one of the maintainers of Zstd.)
    comment by felixhandte · 2025-09-15 · in Removing newlines in FASTA file increases ZSTD compression ratio by 10x
  9. …pple.com/documentation/compression/compre... "COMPRESSION_LZFSE | Apple Developer Documentation" [3]: https://github.com/facebook/zstd?utm_source=chatgpt.com "facebook/zstd: Zstandard - Fast real-time compression ..."
    comment by yelirekim · 2025-08-10 · in Yet Another LLM Rant
  10. …in most popular languages. The same cannot be said about ZSTD with very few implementations except https://github.com/facebook/zstd I'm not even sure there is a good pure Java (no JNI) and Go (without Cgo) implementations for ZSTD. And it…
    comment by citrin_ru · 2025-08-05 · in Consider using Zstandard and/or LZ4 instead of Deflate
  11. …of `pigz`[1] or just a windows build of zstd[2]. [1] https://cosmo.zip/pub/cosmos/tiny/pigz [2] https://github.com/facebook/zstd/releases/latest/
    comment by ZoomZoomZoom · 2025-06-23 · in I ported pigz from Unix to Windows
  12. …zstd. As I understand it, all seekable-zstd are valid zstd, so it should be possible? https://github.com/facebook/zstd?tab=readme-ov-file#the-case...
    comment by ikawe · 2025-06-16 · in Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format
  13. …with normal Zstd decompressors (the seek table is just treated as metadata and ignored). https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
    comment by teraflop · 2025-06-16 · in Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format
  14. …databases transparently, but tool support was kinda lacking. Will the zstd CLI ever support it? https://github.com/facebook/zstd/issues/2121
    comment by ncruces · 2025-06-16 · in Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format
  15. Looking at the spec ( https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... ), I don't see any mention of custom dictionaries like you describe. The spec does mention: > While only Checksum_Flag currently exists, there are 7…
    comment by jeroenhd · 2025-06-16 · in Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format
  16. ▲ 214
    Show HN: Zeekstd – Rust Implementation of the ZSTD Seekable Format
    story by rorosen · 2025-06-15 · 46 comments
  17. …you an error and you don't handle it, the next calls may cause UB, so it kinda does both things. https://github.com/facebook/zstd/blob/b16d193512d3ded82fd584...
    comment by erk__ · 2025-03-08 · in Rust inadequate for text compression codecs?
  18. ▲ 4
    Zstandard v1.5.7 brings performance enhancements
    story by pjf · 2025-02-20 · 1 comment
  19. https://github.com/facebook/zstd/releases/tag/v1.5.7
    comment by pella · 2025-02-20 · in ZSTD 1.5.7 Pushing Compression Performance Even Further
  20. ▲ 5
    Zstandard v1.5.7
    story by pella · 2025-02-19 · 0 comments
  21. …which indicates that the results are outdated - currently, version 1.5.6 is the latest release ( https://github.com/facebook/zstd/releases/tag/v1.5.6 ) In my opinion, it is better to check the original repository:…
    comment by pella · 2025-02-14 · in Lzbench compression benchmark
  22. Yeah, not sure what the original commenter is complaining about. I did "git clone https://github.com/facebook/zstd ", "cd zstd", "make" and have a working binary. Doesn't get easier than that.
    comment by jrockway · 2025-02-01 · in Bzip3: A spiritual successor to BZip2
  23. …select, it will output byte for byte identical results. See a direct answer to this question here: https://github.com/facebook/zstd/issues/2079 I believe all of their compressors are similarly deterministic regardless of number of threads,…
    comment by DannyBee · 2025-01-28 · in My failed attempt to shrink all NPM packages by 5%
  24. …unlike Elon Musk, they are on the record of having some contributions to society as a whole. [1] https://github.com/facebook/zstd
    comment by A4ET8a8uTh0_v2 · 2025-01-27 · in DeepSeek releases Janus Pro, a text-to-image generator [pdf]
  25. …other compression schemes. Having an explicit standard for indices is definitely a plus though. [1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
    comment by lifthrasiir · 2024-09-27 · in New standards for a faster and more private Internet
  26. …need any help using these, or have any questions, please open an issue on Zstandard's GitHub! [0] https://github.com/facebook/zstd/releases/tag/v1.5.6
    comment by terrelln · 2024-09-26 · in New standards for a faster and more private Internet
  27. agree, - in the benchmark "zstd 0.6.0" ( Apr 13, 2016 ) - vs. latest zstd v1.5.6 ( Mar 30, 2024 https://github.com/facebook/zstd/releases )
    comment by pella · 2024-09-20 · in Large Text Compression Benchmark
  28. …point of 7zip a slower setting with better compression ratio is likely perfectly fine. [0] https://github.com/facebook/zstd/tree/dev?tab=readme-ov-file...
    comment by vanderZwan · 2024-08-07 · in Porting my JavaScript game engine to C for no reason
  29. …That seems like an odd claim, unless it used to be a personal project and Facebook took it over ( https://github.com/facebook/zstd/blob/dev/CONTRIBUTING.md#co... )
    comment by lucb1e · 2024-06-13 · in MLow: Meta's low bitrate audio codec
  30. …"small string" dictionary based on representative training data precisely for this use-case. [1] https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md > those are used for different scenarios. zstd is used for data compression, the…
    comment by vlovich123 · 2024-05-07 · in Meta String: A more space-efficient string encoding than UTF-8 in Fury
  31. …If it does, you'd need to actually show this rather than try to argue from first principles. [1] https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md
    comment by vlovich123 · 2024-05-07 · in Meta String: A more space-efficient string encoding than UTF-8 in Fury
  32. …performance would drop for realistic use. But there are also some knobs you can tune. Refer to: https://github.com/facebook/zstd/#dictionary-compression-how... https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin... $ man zstd -…
    comment by Intralexical · 2024-05-05 · in Using a LLM to compress text
  33. This VFS will read a sqlite file after it has been compressed using [zstd seekable format]( https://github.com/facebook/zstd/blob/dev/contrib/seekable_f... ). Built to support read-only databases for full-text search. Benchmarks are…
    comment by sharinglinks123 · 2024-04-27 · in SQLite VFS for ZSTD seekable format
  34. …with zstd if you really want to. It's even in the zstd repo, under a "contrib" implementation https://github.com/facebook/zstd/blob/87af5fb2df7c68cc70c090... That does, of course, require that you compress it into multiple frames to begin…
    comment by dralley · 2024-04-10 · in Bzip2 Format Specification (2016) [pdf]
  35. …decompression natively but hasn't prioritized it given the complexity and lack of immediate need. https://github.com/facebook/zstd/issues/2499#issuecomment-78...
    comment by dralley · 2024-04-10 · in Bzip2 Format Specification (2016) [pdf]
  36. …faster. NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay…
    comment by idoubtit · 2024-04-01 · in Chrome Feature: ZSTD Content-Encoding
  37. …this with a bunch of links to discussions about it, but they continue to not have time it seems. https://github.com/facebook/zstd/issues/3100 This was the first place my mind went when I saw this Content-Encoding announcement, so I ran and…
    comment by saurik · 2024-04-01 · in Chrome Feature: ZSTD Content-Encoding
  38. Citation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75
    comment by cipherboy · 2024-04-01 · in Chrome Feature: ZSTD Content-Encoding
  39. Yes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors , 300+ other contributors have made 4500+ commits to the zstd repo. It's not quite as small-scale as 90's-style shareware…
    comment by KingMob · 2024-04-01 · in Chrome Feature: ZSTD Content-Encoding
  40. details please? I do not see any such contributions to https://github.com/facebook/zstd
    comment by joeyh · 2024-03-29 · in Backdoor in upstream xz/liblzma leading to SSH server compromise
  41. ▲ 4
    Zstd v1.5.6 – Chrome Edition
    story by mfiguiere · 2024-03-27 · 0 comments
  42. …In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.
    comment by def-lkb · 2024-03-09 · in Optimizing Rabin-Karp Hashing
  43. …in zig and I have zstd directly from zon as a dependency. .dependencies = .{ .zstd = .{ .url = "git+https://github.com/facebook/zstd.git#v1.5.5", .hash = "1220185ad79a437fd9f148d1422ff756287534c79a0712105039b4034031480e41a9", }, }, You…
    comment by Cloudef · 2024-02-18 · in Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
  44. I've done that experiment with zstd before. https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md... Not sure about brotli though.
    comment by anonacct37 · 2023-10-04 · in Honey, I shrunk the NPM package
  45. …works (not that a partial archive has no errors, for example). Anyway, details of the bug are here: https://github.com/facebook/zstd/pull/3517 > This [affects] the block splitter. Only levels using the optimal parser (levels 13 and above…
    comment by lucb1e · 2023-04-04 · in ZSTD 1.5.5 is released with a corruption fix found at Google
  46. ▲ 170
    ZSTD 1.5.5 is released with a corruption fix found at Google
    story by danlark · 2023-04-04 · 23 comments
  47. ▲ 74
    Zstandard v1.5.4
    story by ascom · 2023-02-10 · 9 comments
  48. …[5] https://clickhouse.com/docs/en/sql-reference/statements/crea... [6] https://github.com/facebook/zstd/ [7] https://arxiv.org/pdf/2204.09795.pdf "SciTS: A Benchmark for Time-Series Databases in Scientific Experiments and Industr
    comment by zX41ZdbW · 2023-02-01 · in Float Compression 3: Filters
  49. …which creates a file that stores the "patch" to create a new file from an old (reference) file. See https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin... See the man page: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.…
    comment by wolf550e · 2023-01-09 · in We're wasting money by only supporting gzip for raw DNA files
  50. …make their own decisions on who to ban or not, and which are all manual. Just as you can comment on, say, github.com/facebook/zstd without a Facebook account, you can also comment on github.com/golang/go without a Google account.
    comment by Beltalowda · 2023-01-09 · in Sourcehut will blacklist the Go module mirror