Hacker GitHub

← All repos

mxmlnkn/rapidgzip

View on GitHub ↗ ★ 472

Gzip Decompression and Random Access for Modern Multi-Core Machines

16 mentions on Hacker News by 3 people: 4 stories, 12 comments · first 2022-08-06 · latest 2024-09-18

Mentions per year 16 mentions · busiest year 2023 (11)
20082026
AllStoriesComments NewestTopOldest
  1. …you're dealing with multi-TB files, this is a big deal. [0]: https://github.com/madler/pigz [1]: https://github.com/mxmlnkn/rapidgzip
    comment by sgarland · 2024-09-18 · in Dissecting the gzip format (2011)
  2. ▲ 35
    Show HN: Rapidgzip – Truly Parallel Gzip Decompression with 10 GB/s
    story by mxmlnkn · 2023-09-04 · 12 comments
  3. …speed of the gzip command line utility, which for some weird reason is half as fast as zlib. [0] https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchma...
    comment by mxmlnkn · 2023-08-30 · in Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
  4. …the test case and the one I tested to, too quickly, check my statement is one of the outliers. [0] https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2...
    comment by mxmlnkn · 2023-08-30 · in Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
  5. …decompressor. A probably incomplete list of such implementations can be found in this issue: https://github.com/mxmlnkn/rapidgzip/issues/8 The index makes it so easy that I can simply delegate decompression to zlib. And since paper…
    comment by mxmlnkn · 2023-08-21 · in Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf]
  6. Repo: https://github.com/mxmlnkn/rapidgzip As someone who has dabbled quite a lot with deflate this was a very interesting find. It seems like the format that never wants to die. First of all I am surprised this is even possible. Given the…
    comment by klauspost · 2023-08-21 · in Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf]
  7. ▲ 4
    Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf]
    story by klauspost · 2023-08-21 · 2 comments
  8. …though, as I have no QAT-enabled device. [1] https://github.com/intel/isa-l/tree/master/igzip [2] https://github.com/mxmlnkn/rapidgzip [3] https://github.com/intel/QATzip#test-qatzip
    comment by mxmlnkn · 2023-08-17 · in Intel QuickAssist Technology Zstandard Plugin for Zstandard
  9. ▲ 3
    Pragzip: Tool and Library for Parallel Gzip Decompression and Random Access
    story by mxmlnkn · 2023-05-12 · 0 comments
  10. …not only implemented parallel decompression but also random access to offsets in the stream with https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able to reach over 10 GB/s…
    comment by mxmlnkn · 2023-05-12 · in Pigz: Parallel gzip for modern multi-processor, multi-core machines
  11. There also is this: https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able to reach almost 20 GB/s decompression bandwidth.
    comment by mxmlnkn · 2023-05-12 · in Pigz: Parallel gzip for modern multi-processor, multi-core machines
  12. Decompression of arbitrary gzip files can be parallelized with pragzip: https://github.com/mxmlnkn/pragzip
    comment by mxmlnkn · 2023-04-04 · in Parquet: An efficient, binary file format for table data
  13. …read calling loop. Here is the full commit with ad-hoc benchmark results in the commit message: https://github.com/mxmlnkn/pragzip/commit/0b1af498377838c30f... and here the benchmarks I ran at that time:…
    comment by mxmlnkn · 2022-11-13 · in The Cost of Exception Handling
  14. I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.
    comment by mxmlnkn · 2022-10-18 · in pigz: A parallel implementation of gzip for multi-core machines
  15. …in parallel in a similar manner to the prototype pugz by implementing a two-staged decompression. https://github.com/mxmlnkn/pragzip I also did a Show HN: https://news.ycombinator.com/item?id=32366959
    comment by mxmlnkn · 2022-10-11 · in How much faster is making a tar archive without gzip?
  16. ▲ 15
    Show HN: Thread-Parallel Decompression and Random Access to Gzip Files (Pragzip)
    story by mxmlnkn · 2022-08-06 · 3 comments