mxmlnkn/rapidgzip
View on GitHub ↗ ★ 472
Gzip Decompression and Random Access for Modern Multi-Core Machines
16 mentions on Hacker News by 3 people: 4 stories, 12 comments · first 2022-08-06 · latest 2024-09-18
20082026
-
…you're dealing with multi-TB files, this is a big deal. [0]: https://github.com/madler/pigz [1]: https://github.com/mxmlnkn/rapidgzip
comment by sgarland · 2024-09-18 · in Dissecting the gzip format (2011) -
…speed of the gzip command line utility, which for some weird reason is half as fast as zlib. [0] https://github.com/mxmlnkn/rapidgzip/blob/master/src/benchma...
comment by mxmlnkn · 2023-08-30 · in Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library -
…the test case and the one I tested to, too quickly, check my statement is one of the outliers. [0] https://github.com/mxmlnkn/rapidgzip#scaling-benchmarks-on-2...
comment by mxmlnkn · 2023-08-30 · in Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library -
…decompressor. A probably incomplete list of such implementations can be found in this issue: https://github.com/mxmlnkn/rapidgzip/issues/8 The index makes it so easy that I can simply delegate decompression to zlib. And since paper…
comment by mxmlnkn · 2023-08-21 · in Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf] -
Repo: https://github.com/mxmlnkn/rapidgzip As someone who has dabbled quite a lot with deflate this was a very interesting find. It seems like the format that never wants to die. First of all I am surprised this is even possible. Given the…
comment by klauspost · 2023-08-21 · in Rapidgzip – Parallel Decompression and Seeking in Gzip (Knespel, Brunst – 2023) [pdf] -
…though, as I have no QAT-enabled device. [1] https://github.com/intel/isa-l/tree/master/igzip [2] https://github.com/mxmlnkn/rapidgzip [3] https://github.com/intel/QATzip#test-qatzip
comment by mxmlnkn · 2023-08-17 · in Intel QuickAssist Technology Zstandard Plugin for Zstandard -
…not only implemented parallel decompression but also random access to offsets in the stream with https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able to reach over 10 GB/s…
comment by mxmlnkn · 2023-05-12 · in Pigz: Parallel gzip for modern multi-processor, multi-core machines -
There also is this: https://github.com/mxmlnkn/pragzip I did some benchmarks on some really beefy machines with 128 cores and was able to reach almost 20 GB/s decompression bandwidth.
comment by mxmlnkn · 2023-05-12 · in Pigz: Parallel gzip for modern multi-processor, multi-core machines -
Decompression of arbitrary gzip files can be parallelized with pragzip: https://github.com/mxmlnkn/pragzip
comment by mxmlnkn · 2023-04-04 · in Parquet: An efficient, binary file format for table data -
…read calling loop. Here is the full commit with ad-hoc benchmark results in the commit message: https://github.com/mxmlnkn/pragzip/commit/0b1af498377838c30f... and here the benchmarks I ran at that time:…
comment by mxmlnkn · 2022-11-13 · in The Cost of Exception Handling -
I recently implemented pragzip for parallel gzip decompression https://github.com/mxmlnkn/pragzip . I would be interested to know how your PR back then worked to parallelize the decompression.
comment by mxmlnkn · 2022-10-18 · in pigz: A parallel implementation of gzip for multi-core machines -
…in parallel in a similar manner to the prototype pugz by implementing a two-staged decompression. https://github.com/mxmlnkn/pragzip I also did a Show HN: https://news.ycombinator.com/item?id=32366959
comment by mxmlnkn · 2022-10-11 · in How much faster is making a tar archive without gzip?