cube2222/octosql on Hacker News

Octosql does this as well as a few other formats. I've found it useful several times. https://github.com/cube2222/octosql

comment by nbk_2000 · 2024-10-07 · in Sq.io: jq for databases and more

…creators good luck, this looks great! [0]: https://materialize.com [1]: https://www.epsio.io [2]: https://github.com/cube2222/octosql [3]: https://www.scattered-thoughts.net/writing/internal-consiste... [4]: https://arxiv.org/pdf/2203.16684

comment by cube2222 · 2024-09-29 · in Feldera Incremental Compute Engine

OctoSQL[1] does a pretty good job of allowing you to query JSON (and CSV) with SQL. [1] https://github.com/cube2222/octosql

comment by nbk_2000 · 2023-11-30 · in Jaq – A jq clone focused on correctness, speed, and simplicity

…the guy from my team who is the acting TL of the OpenTF project is the creator and maintainer of https://github.com/cube2222/octosql

comment by fishnchips · 2023-08-25 · in OpenTF announces fork of Terraform

…prototype in llvm and then cranelift. That said, here's some of the wazero-based code on a branch - https://github.com/cube2222/octosql/tree/wasm-experiment/was... It really is just a very very basic prototype.

comment by cube2222 · 2023-07-01 · in Wazero: Zero dependency WebAssembly runtime written in Go

…I've enjoyed using for this. It parses CSV and JSON, which are the file types I parse the most. [1] https://github.com/cube2222/octosql/

comment by nbk_2000 · 2023-06-28 · in Using command line to process CSV files (2022)

…curious about the details here) [0]: https://github.com/TimelyDataflow/differential-dataflow [1]: https://github.com/cube2222/octosql

comment by cube2222 · 2023-06-07 · in Hydroflow: Dataflow Runtime in Rust

…while going through 10 GB of JSON in a couple seconds at most. Disclaimer: author of OctoSQL [0]: https://github.com/cube2222/octosql [1]: https://duckdb.org/

comment by cube2222 · 2023-03-18 · in Analyzing multi-gigabyte JSON files locally

…for various things. Godspeed! [0]: https://duckdb.org/2022/09/30/postgres-scanner.html [1]: https://github.com/cube2222/octosql [2]: https://steampipe.io

comment by cube2222 · 2023-03-03 · in DuckDB: Querying JSON files as if they were tables

Very interesting, sounds kinda like GraphQL counterpart to https://github.com/cube2222/octosql

comment by sdfhbdf · 2023-02-09 · in Trustfall: How to Query (Almost) Everything

…processing itself being ridiculously fast). Godspeed with the future evolution of ClickHouse! [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2023-01-05 · in Show HN: ClickHouse-local – a small tool for serverless data analytics

…[0]: https://github.com/apache/arrow-datafusion [1]: https://github.com/multiprocessio/dsq [2]: https://github.com/cube2222/octosql Disclaimer: Author of OctoSQL

comment by cube2222 · 2022-11-03 · in Command-line data analytics

…"compute" part of the "next gen Materialize"? Congrats and good luck with further development! [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-10-03 · in The next generation of Materialize

…to more than just DevOps data. I've used the benchmark available in the OctoSQL README. [0]: https://github.com/cube2222/octosql [1]: https://github.com/apache/arrow-datafusion Disclaimer: author of OctoSQL

comment by cube2222 · 2022-09-30 · in Select * from cloud

…(JIT-ish). [0]: https://github.com/hashicorp/terraform [1]: https://github.com/traefik/traefik [2]: https://github.com/cube2222/octosql Disclaimer: author of OctoSQL

comment by cube2222 · 2022-09-26 · in Go runtime: 4 years later

…Two alternative CLI tools that I looked at: sq - https://sq.io/ octosql - https://github.com/cube2222/octosql Honourable mentions to: Miller - https://miller.readthedocs.io/en/latest/index.html Dasel - https://daseldocs.tomwright.me

comment by sjellis · 2022-09-25 · in Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

…row table, it'll just get those 10 rows instead of getting them all and filtering in memory. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-09-24 · in Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

…much pieces you have to specify if you're not an artist - you don't know what you don't know. [0]: https://github.com/cube2222/octosql [1]: http://dallery.gallery/wp-content/uploads/2022/07/The-DALL%C...

comment by cube2222 · 2022-08-02 · in I used DALL·E 2 to generate a logo

…But anyhow, congrats on the project, and I'm pumped to see what you come up with next! [0]: https://github.com/cube2222/octosql [1]: https://github.com/dcmoura/spyql/blob/master/notebooks/json_...

comment by cube2222 · 2022-07-30 · in A SQLite extension for reading large files line-by-line

The docs panic issue has now been fixed[0] and released[1]. Thanks again for mentioning it! [0]: https://github.com/cube2222/octosql/commit/f7a1b92ba65584bc6... [1]: https://github.com/cube2222/octosql/releases/tag/v0.8.1

comment by cube2222 · 2022-07-17 · in OctoSQL allows you to join data from different sources using SQL

…and now a query like this should once again take just ~tens of milliseconds to complete. [0]: https://github.com/cube2222/octosql/commit/ba4a8896be35ffdcc...

comment by cube2222 · 2022-07-17 · in OctoSQL allows you to join data from different sources using SQL

…also LOOKUP JOIN with the `plugins.available_plugins` table if that is indeed what you want). [0]: https://github.com/cube2222/octosql-plugin-random_data [1]: https://steampipe.io [2]: https://github.com/cube2222/octosql/blob/main/datasourc…

comment by cube2222 · 2022-07-16 · in OctoSQL allows you to join data from different sources using SQL

…the rest will get handled by the SDK. Here's an example plugin that simple and fairly easy to read: https://github.com/cube2222/octosql-plugin-random_data And here's the gRPC schema for plugins, which might be useful to get a better…

comment by cube2222 · 2022-07-16 · in OctoSQL allows you to join data from different sources using SQL

…intensive and problematic the GC really is, even though reading it takes some getting used to. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-07-15 · in A Guide to the Go Garbage Collector

…argument syntax error at position 13 near 'index' Hope this helps and I wish you all the best. [1] https://github.com/cube2222/octosql/wiki/Function-Documentat...

comment by aargh_aargh · 2022-06-21 · in One-liner for running queries against CSV files with SQLite

…You can use it with csv, json, parquet but also Postgres, MySQL, etc. All in a single query! [0]: https://github.com/cube2222/octosql Disclaimer: author of OctoSQL

comment by cube2222 · 2022-06-21 · in One-liner for running queries against CSV files with SQLite

…solution (like BigQuery). Do you have any thoughts on that? [0]: https://datasette.io [1]: https://github.com/cube2222/octosql [2]: https://steampipe.io

comment by cube2222 · 2022-06-21 · in One year as a solo dev building open-source data tools without funding

…sending aggregated telemetry data every 10 invocations (as well as on the very first one). [0]: https://github.com/cube2222/octosql [1]: https://github.com/cube2222/octosql/blob/main/telemetry/tele...

comment by cube2222 · 2022-05-28 · in GitHub Star History Graph

…Both resulted in substantial changes to the overall picture painted by the benchmarks. [0]: https://github.com/cube2222/octosql [1]: https://news.ycombinator.com/item?id=31004563

comment by cube2222 · 2022-04-21 · in The fastest tool for querying JSON files is *not* written in Python (revised)

…buffering and sorting them) does indeed run 2-3x faster and use 50 MB RAM - instead of GB's. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-04-15 · in The fastest tool for querying large JSON files is written in Python (benchmark)

…per second. That said, compiling with Go 1.18 vs Go 1.17 got me a 10-15% speedup by itself. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-03-30 · in Generics can make your Go code slower

[0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-01-26 · in Show HN: SPyQL – SQL with Python in the middle

> [0]: https://github.com/cube2222/octosql nice tool, thanks for making this available.

comment by sugarkjube · 2022-01-19 · in Tricking PostgreSQL into using an insane, but faster, query plan

…I've implemented - COUNT(*) is able to optimize out the most underlying data loading. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2022-01-18 · in Tricking PostgreSQL into using an insane, but faster, query plan

…for a lot more supported datasources, as the plugin system lets me work on that much quicker. [0]: https://github.com/cube2222/octosql/releases/tag/v0.4.0 [1]: https://github.com/cube2222/octosql#readme

comment by cube2222 · 2022-01-11 · in Dsq: Commandline tool for running SQL queries against JSON, CSV, Parquet, etc.

…https://github.com/harelba/q [1]: https://github.com/dinedal/textql [2]: https://datasette.io [3]: https://github.com/cube2222/octosql

comment by cube2222 · 2021-07-18 · in Show HN: Work with CSV files using SQL. For data scientists and engineers

…using temporal SQL extensions. (Inspired by the great paper, One SQL To Rule Them All[1]) [0]: https://github.com/cube2222/octosql [1]: https://arxiv.org/abs/1905.12133

comment by cube2222 · 2021-06-07 · in Q – Run SQL Directly on CSV or TSV Files

…also delves into details why the push-based architecture has more room for optimizations. [0]: https://github.com/cube2222/octosql/tree/redesign [1]: https://github.com/cube2222/octosql/

comment by cube2222 · 2021-05-01 · in Query Engines: Push vs. Pull

…which gets rid of wrong decisions and makes it easier to embed in existing applications. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2021-02-16 · in Minimally Invasive (and More Accurate) Analytics: GoAccess and Athena/SQL

…different use case, but you should check it out nevertheless! The naming clash is funny. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-09-11 · in Show HN: Octo – Generate a serverless API from an SQL query

…Excel and Parquet files. It also let's you join them with each other (and other databases). [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-08-16 · in Using Askgit – A SQL interface to your Git repository

…what users actually use. Please let us know what you think! Feedback is greatly appreciated :) [0]: https://github.com/cube2222/octosql/blob/master/whitepapers/...

comment by cube2222 · 2020-08-11 · in OctoSQL Streaming: Adding Temporal SQL on Streams

…just create a tuple, or tuple of tuples out of it, which does act like a single value. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-08-11 · in We Can Do Better Than SQL

…a more detailed description of implementation details, tradeoffs and comparisons to other projects: https://github.com/cube2222/octosql/raw/master/whitepapers/t... BTW If you like what octosql is doing and would like to help speed up…

comment by cube2222 · 2020-08-05 · in OctoSQL v0.3.0: Streaming, Temporal SQL on Event Streams in Go

…out, it's also very simple to get going locally! Disclosure: I'm one of the main contributors. [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-05-20 · in Apache Drill's Future

…at least the storage engine they're using is awesome, and I can definitely recommend it! [1]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-05-01 · in Jepsen: Dgraph 1.1.1

Indeed, I remember! :) The parquet support is mostly ready as far as I know here https://github.com/cube2222/octosql/pull/153/ but I haven't tested it myself yet. However, we'll surely have it ready for the "streaming" release which is…

comment by cube2222 · 2020-04-13 · in Writing a SQL database from scratch in Go

…is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-04-13 · in Writing a SQL database from scratch in Go

…dataflow-like retraction model for that. We should have a big release out soon, so stay tuned! [1]: https://github.com/cube2222/octosql

comment by cube2222 · 2020-03-21 · in Ask HN: What projects are you working on now?

…I think they all are fascinating! This is a big area with lots of space for innovation left.) [1]: https://github.com/cube2222/octosql [2]: https://github.com/cube2222/jql

comment by cube2222 · 2020-02-25 · in Show HN: Querying JSON documents using SQL-like language in Scala