madler/pigz
View on GitHub ↗ ★ 2,975 C
A parallel implementation of gzip for modern multi-processor, multi-core machines.
11 mentions on Hacker News by 10 people: 2 stories, 9 comments · first 2016-08-20 · latest 2024-09-18
20082026
-
…(rapidgzip [1]), for one. When you're dealing with multi-TB files, this is a big deal. [0]: https://github.com/madler/pigz [1]: https://github.com/mxmlnkn/rapidgzip
comment by sgarland · 2024-09-18 · in Dissecting the gzip format (2011) -
…That's very interesting, and perhaps suggests that bgzip is a tool you would be interested in. 2. https://github.com/madler/pigz/pull/19
comment by mgerdts · 2023-05-12 · in Pigz: Parallel gzip for modern multi-processor, multi-core machines -
…compression prevent that failure case, at least limiting it to its block size of a default 128K? https://github.com/madler/pigz/blob/master/pigz.1#L43-L45 That would be a nice extra benefit, besides the speedup from being multithreaded. (I…
comment by lucb1e · 2023-04-04 · in ZSTD 1.5.5 is released with a corruption fix found at Google -
…is, you don't need to write your own compressor or rely on block alignment, because: • pigz ( https://github.com/madler/pigz ) has -0 for no-compression compression; • you can concat multiple gzip files together. $ echo foo | gzip >…
comment by jwilk · 2022-12-04 · in Creating a modifiable gzipped disk image -
The bit I found most interesting was actually: https://github.com/madler/pigz/blob/master/try.h https://github.com/madler/pigz/blob/master/try.c which implements try/catch for C99.
comment by omoikane · 2022-10-17 · in pigz: A parallel implementation of gzip for multi-core machines -
▲ 289
pigz: A parallel implementation of gzip for multi-core machinesstory by firloop · 2022-10-17 · 106 comments
-
…of benchmarking things — especially when you're only considering them for performance reasons. See: https://github.com/madler/pigz/issues/36 For compression, pigz is only useful for compressing a single large file in a scenario where you…
comment by goodside · 2020-07-12 · in Parallel Gzip – Pigz -
…. From the readme: "Contrary to the pigz program which does single-threaded decompression (see https://github.com/madler/pigz/blob/master/pigz.c#L232 ), pugz found a way to do truly parallel decompression. In a nutshell: the compressed…
comment by nkrumm · 2019-05-20 · in Parallel decompression of gzip-compressed files -
pigz doesn't help much with decompression: https://github.com/madler/pigz/issues/36#issuecomment-249041... Decompression can’t be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz…
comment by jwilk · 2018-07-31 · in How F5Bot Slurps All of Reddit -
…details of the design of pigz, there is a very well-documented overview in the source of pigz.c: https://github.com/madler/pigz/blob/master/pigz.c#L187
comment by kbaker · 2016-08-20 · in A parallel implementation of gzip for modern multi-processor multi-core machines -
▲ 280
A parallel implementation of gzip for modern multi-processor multi-core machinesstory by odeke-em · 2016-08-20 · 67 comments