liquidaty/zsv on Hacker News

Haven't yet seen any of these beat https://github.com/liquidaty/zsv (of which I'm an author) when real-world constraints are applied (e.g. we no longer assume that line ends are always \n, or that there are no dbl-quote chars, embedded…

comment by mattewong · 2024-10-28 · in How fast can you parse a CSV file in C#?

…it is worth anyone's time to follow this article's suggestion vs just using something like zsv ( https://github.com/liquidaty/zsv , which I'm an author of) or xsv ( https://github.com/BurntSushi/xsv/edit/master/README.md ) and then…

comment by mattewong · 2024-08-14 · in CSVs Are Kinda Bad. DSVs Are Kinda Good

…or CSV to make use of high-performance tools like xsv or zsv (the latter of which I'm an author). https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql... https://github.com/BurntSushi/xsv

comment by mattewong · 2023-03-19 · in Analyzing multi-gigabyte JSON files locally

…and miller are both extraordinarily slow try xsv ( https://github.com/BurntSushi/xsv ) or zsv ( https://github.com/liquidaty/zsv ) instead (the latter of which I'm an author of)

comment by mattewong · 2023-03-14 · in Ask HN: What modern tools should be standard part of a modern unixy distro?Why?

Parsing CSV doesn't have to be slow if you use something like xsv or zsv ( https://github.com/liquidaty/zsv ) (disclaimer: I'm an author). The speed of CSV parsers is fast enough that unless you are doing something ultra-trivial such as…

comment by mattewong · 2023-03-07 · in Show HN: Faster FastAPI with simdjson and io_uring on Linux 5.19

Another for the mix: https://github.com/liquidaty/zsv

comment by mattewong · 2023-02-05 · in yq: command-line YAML, JSON, XML, CSV and properties processor

…more I used it, the more technical debt I accumulated. This was one of the reasons I wrote zsv ( https://github.com/liquidaty/zsv ). Maybe csvkit could incorporate the zsv engine and we could get the best of both worlds? Examples (using…

comment by mattewong · 2023-01-20 · in csvkit: Command-line tools for working with CSV

…https://github.com/BurntSushi/xsv is much faster than mlr (like an order of magnitude), and zsv ( https://github.com/liquidaty/zsv ) is even faster. But, neither support formulas. Disclaimer: I am one of the zsv authors

comment by mattewong · 2022-12-21 · in Ask HN: Programs that saved you 100 hours? (2022 edition)

https://github.com/liquidaty/zsv

comment by mattewong · 2022-12-21 · in Ask HN: Programs that saved you 100 hours? (2022 edition)

…For the preprocessing, you can use something like the `2tsv` or `select -e ...` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author) to ensure each record is a single line. You can also use something like `xsv split`…

comment by mattewong · 2022-10-18 · in Show HN: An app to split CSV into multiple files to avoid Excel's 1M row limit

another shameless plug: https://github.com/liquidaty/zsv

comment by mattewong · 2022-09-26 · in Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

…two seconds or so to load, and less than a second to run, and gives me the correct result. Using https://github.com/liquidaty/zsv (disclaimer, I'm one of its authors), I get the correct results in 0.95 seconds with the one-liner `zsv sql…

comment by mattewong · 2022-09-22 · in Q – Run SQL Directly on CSV or TSV Files

…when you can just push format X through a pipe and get whichever format Y you want back out: https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql... (disclaimer: I'm one of the zsv authors)

comment by mattewong · 2022-09-08 · in Health insurers just published close to a trillion hospital prices

…1mm-row worldcitiespop_mil file, q takes 27 seconds compared to `zsv sql` which takes 1.7 seconds ( https://github.com/liquidaty/zsv ) and also supports multiple file joins. I'm sure q is faster once cached, but taking a 16x performance…

comment by mattewong · 2022-06-23 · in One-liner for running queries against CSV files with SQLite

https://github.com/liquidaty/zsv/blob/main/app/external/sqli... modifies the sqlite3 virtual table engine to use the faster zsv parser. have not quantified the difference, but in all tests I have run, `zsv sql` runs faster (sometimes much…

comment by mattewong · 2022-06-21 · in One-liner for running queries against CSV files with SQLite

Over all these CLIs, I prefer zsv ( https://github.com/liquidaty/zsv )-- then again, I wrote it so my preference should come as no surprise. On my Mac was 340% faster than the OP command, and does a lot more than just SQL/sqlite3-related

comment by mattewong · 2022-06-21 · in One-liner for running queries against CSV files with SQLite

…(or in other ways) with https://github.com/BurntSushi/xsv , https://github.com/eBay/tsv-utils or https://github.com/liquidaty/zsv ? (disclaimer: I wrote zsv)

comment by mattewong · 2022-06-21 · in Splitting CSV files at 3GB/s

…quite slow, literally off-the-charts slow for our purposes when we benchmarked vs xsv and zsv (see https://github.com/liquidaty/zsv . disclaimer: I'm one of its authors)

comment by mattewong · 2022-05-12 · in Modernizing AWK, a 45-year old language, by adding CSV support

…(credit to xsv for choosing that dataset) against https://github.com/BurntSushi/xsv and https://github.com/liquidaty/zsv (full disclosure: I am one of the zsv authors). Here's what I got. fastest to slowest: zsv (0.07), xsv (0.16), goawk…

comment by mattewong · 2022-05-11 · in Modernizing AWK, a 45-year old language, by adding CSV support

…comfortably into memory 2. Sort each shard. You could use something like `zsv` (sql sub-command) ( https://github.com/liquidaty/zsv ) or `xsv` (sort subcommand) ( https://github.com/BurntSushi/xsv ) for this 3. Use mergesort to sort all…

comment by mattewong · 2022-04-18 · in Ask HN: How to Deal with CSVs?

…subset of the features there are available in a native command-line utility (all open-source-- see https://github.com/liquidaty/zsv ), to which the "obfuscate" feature that is currently available in the playground will soon be available…

comment by mattewong · 2022-01-29 · in Show HN: Random Data Generator for arbitrary data types

…(needs the ending slash) which is powered by a web version of the SIMD parser ( https://github.com/liquidaty/zsv ). BTW some quick performance notes on the zsv CLI (not the playground): prelim testing is ~2x faster than xsv or tsv-utils o

comment by mattewong · 2021-12-21 · in Show HN: Full-featured+fast SIMD CSV lib, extensible utility & web playground

As promised: https://github.com/liquidaty/zsv

comment by mattewong · 2021-12-20 · in Leveraging SIMD: Splitting CSV Files at 3Gb/S