google/riegeli
View on GitHub ↗ ★ 496 C++
Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers.
9 mentions on Hacker News by 8 people: 0 stories, 9 comments · first 2018-04-11 · latest 2026-09-03
20082026
-
…bottleneck. It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well,…
comment by chickenbig · 2026-09-03 · in We could save petabytes of cache storage with Zstandard and Pingora -
Even compression level 1 or 2 is pretty good. I once used https://github.com/google/riegeli and a low zstd compression level to store large quantities of protobuf data in an efficient manner (in terms of CPU, RAM and streaming to disk).…
comment by chickenbig · 2026-06-27 · in Hellishly Slow Level 13 Deflate Compression -
…schemas. They are only selectively open sourced. Have a look at riegeli if you are interested. https://github.com/google/riegeli
comment by squirrellous · 2025-12-03 · in Why I stopped using JSON for my APIs -
…corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli
comment by NavinF · 2023-09-20 · in Add extra stuff to a “standard” encoding? Sure, why not -
…routers, logging, ACL and ratelimit systems up to the actual service handling the request. [2] - https://github.com/google/riegeli
comment by q3k · 2021-10-11 · in Parsing JSON Is a Minefield (2016) -
…on three different compressors and Abseil, and its file format specification is quite elaborate: https://github.com/google/riegeli/blob/master/doc/riegeli_re... . This could explain its lack of market penetration (only 221 stars at the…
comment by m0zg · 2020-06-21 · in FlexBuffers -
It looks like https://github.com/google/riegeli might be what you're looking for? (from a search of "RecordIO")
comment by vore · 2020-06-21 · in FlexBuffers -
…I couldn't find anything. So I implemented https://github.com/romkatv/ChunkIO . I later learned of https://github.com/google/riegeli (work in progress), which could've saved me a lot of time if only I found it earlier. I think my ChunkIO…
comment by romka2 · 2019-04-10 · in Xg2xg: Lookup table of similar tech and services for ex-Googlers -
If you were interested in RecordIO, then this project might also be of interest to you: https://github.com/google/riegeli
comment by vinkelhake · 2018-04-11 · in Prototool – A Swiss Army Knife for Protocol Buffers