apache/datafusion on Hacker News

…Claude Code used for profiling and implementing the performance improvements - apache/datafusion: https://github.com/apache/datafusion/pull/21182 - up to 49x speedup of LIMIT queries by eliminating unnecessary sorts and pushing limits into…

comment by simonw · 2026-08-09 · in Is it all just vapourware?

Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395

comment by stuhood · 2026-08-03 · in Algorithms on billion-scale graph using 10GB RAM: I love DataFusion

You might find https://github.com/apache/datafusion and https://github.com/datafusion-contrib/datafusion-federation of interest

comment by jeadie · 2026-04-14 · in Distributed DuckDB Instance

…SQL layers — GlueSQL as a simpler starting point, DataFusion as the more complete option ( https://github.com/apache/datafusion ). Plugging into a real SQL engine seems like a great strategy for uncovering bugs, fixing correctness issues,…

comment by nlz · 2026-03-05 · in Show HN: I built an LSM storage engine from scratch in Rust

…from signed to unsigned types. I spent a very frustrating day tracking down this one in particular https://github.com/apache/datafusion/issues/15967

comment by thinkharderdev · 2026-02-12 · in Apache Arrow is 10 years old

…arrow.apache.org/docs/format/Columnar.html#format-me... ) I expect this to be used to support Variant https://github.com/apache/datafusion/issues/16116 and geometry types (note I am an author)

comment by alamb · 2025-07-15 · in Embedding user-defined indexes in Apache Parquet

…this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644

comment by jasim · 2025-07-15 · in Embedding user-defined indexes in Apache Parquet

…will outperform them in the long run. Benthos: https://github.com/redpanda-data/benthos DataFusion: https://github.com/apache/datafusion

comment by chenquan · 2025-03-14 · in ArkFlow – High-performance Rust stream processing engine

…to open source, looking for part-time/full-time working with databases. Datafusion contributions: https://github.com/apache/datafusion/pulls?q=is%3Apr+author%... Datafusion comet contributions: https://github.com/apache/datafusion-comet/pul…

comment by jatin510 · 2025-02-04 · in Ask HN: Who wants to be hired? (February 2025)

…Who can build a custom file format that gets the best ClickHouse performance (on DataFusion): https://github.com/apache/datafusion/issues/13448 Disclaimer I am on the PMC of Apache DataFusion, so am totally a fan boy.

comment by alamb · 2024-11-21 · in Building Databases over a Weekend

…[2] https://www.uber.com/blog/modernizing-logging-with-clp-ii/ [3] https://github.com/apache/datafusion

comment by jcgrillo · 2024-09-19 · in OpenTelemetry and vendor neutrality: how to build an observability strategy

…length for compressed given the 255 symbols limit? [1] https://github.com/spiraldb/vortex [2] https://github.com/apache/datafusion

comment by jcgrillo · 2024-09-13 · in Random access string compression with FSST and Rust

…is AGPL v3 + business licensed. [3] [1] https://github.com/duckdb/duckdb/blob/main/LICENSE [2] https://github.com/apache/datafusion/blob/main/LICENSE.txt [3] https://github.com/paradedb/paradedb/blob/dev/LICENSE Most companies won't touch…

comment by nikhilsimha · 2024-06-13 · in Pg_lakehouse: A DuckDB Alternative in Postgres

…the Apache Arrow DataFusion discussion that they too see themselves as a bit of a Velox competitor. https://github.com/apache/arrow-datafusion/discussions/6441 It's cool to see this space mature. I like that even Velox sees that Apache…

comment by jauntywundrkind · 2024-03-25 · in Velox: Meta's Unified Execution Engine [pdf]

…user defined functions, among others) and it implements a substrait provider and consumer: https://github.com/apache/arrow-datafusion/tree/main/datafus...

comment by simicd · 2024-01-29 · in What I talk about when I talk about query optimizer (part 1): IR design

Specifically: https://github.com/apache/arrow-datafusion/tree/main/datafus...

comment by muizelaar · 2024-01-29 · in What I talk about when I talk about query optimizer (part 1): IR design

…progress[0]. The draft[1] includes a comparison to DuckDB and preliminary benchmark results. [0]: https://github.com/apache/arrow-datafusion/issues/6782 [1]: https://www.overleaf.com/read/qjhrxqhgksvr

comment by leicmi · 2023-11-06 · in Updates to the H2O.ai db-benchmark

…date and haven't looked too closely on whether they are keeping it fair for everyone else :) [0]: https://github.com/apache/arrow-datafusion [1]: https://benchmark.clickhouse.com

comment by treesciencebot · 2023-11-06 · in Updates to the H2O.ai db-benchmark

…in DataFusion (part of the Arrow project and built with Arrow as the underlying in-memory format) https://github.com/apache/arrow-datafusion/blob/master/bench... Disclaimer: I'm a committer on the Arrow project and contributor to…

comment by thinkharderdev · 2023-01-10 · in Demystifying Apache Arrow (2020)

…[2] https://github.com/pola-rs/polars/ [3] https://github.com/dimforge/nalgebra [4] https://github.com/apache/arrow-datafusion [5] https://github.com/apache/arrow-ballista

comment by Gwypaas · 2022-12-22 · in Welcome to Comprehensive Rust

…FROM tripdata GROUP BY 1 ORDER BY 1 ASC)` on a 55M row subset of NY Taxi trip data. [0]: https://github.com/apache/arrow-datafusion/blob/master/dataf... [1]: https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0 [2]:…

comment by mildbyte · 2022-11-03 · in Command-line data analytics

…supports normal databases, not just file formats, so you can i.e. join with a Postgres table. [0]: https://github.com/apache/arrow-datafusion [1]: https://github.com/multiprocessio/dsq [2]: https://github.com/cube2222/octosql Disclaimer:…

comment by cube2222 · 2022-11-03 · in Command-line data analytics

…from the user's browser (caching, CDNs, low latency, some WASM support, all that fun stuff). [0] https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-ballista [2] https://github.com/splitgraph/seafowl

comment by mildbyte · 2022-10-26 · in IOx: InfluxData’s New Storage Engine

…recently learned about is Apache DataFusion, also written in rust and uses Arrow memory format: https://github.com/apache/arrow-datafusion/blob/master/READM...

comment by infogulch · 2022-10-23 · in GlueSQL: SQL database engine as a library

…used the benchmark available in the OctoSQL README. [0]: https://github.com/cube2222/octosql [1]: https://github.com/apache/arrow-datafusion Disclaimer: author of OctoSQL

comment by cube2222 · 2022-09-30 · in Select * from cloud

Rust is also being used a lot in that space: https://github.com/MaterializeInc/materialize https://github.com/apache/arrow-datafusion https://github.com/pola-rs/polars

comment by KptMarchewa · 2022-08-12 · in Ask HN: Who is using C++ as the main language for new project?

…has any ideas about where to look, please let me know. [1] https://stripe.com/en-gb/sigma [2] https://github.com/apache/arrow-datafusion

comment by madebyherzblut · 2022-04-22 · in Open Source SQL Parsers

…them/throttling them to be extremely slow to simulate a brownout/straggling cluster node. [1] https://github.com/apache/arrow-datafusion

comment by tanelpoder · 2022-02-27 · in Ask HN: What can I do with 48GB of RAM?

Thanks! It's using Datafusion as the query engine: https://github.com/apache/arrow-datafusion

comment by houqp · 2021-09-16 · in Show HN: Columnq brings OLAP to Unix pipes

…will see uptake and improve this. [0] https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law [1] https://github.com/apache/arrow-datafusion/tree/master/balli...

comment by mynameisash · 2021-09-07 · in My £4 a month server can handle 4.2M requests a day

…added). There is still some missing SQL functionality (for example to run all the TPC-H queries https://github.com/apache/arrow-datafusion/tree/master/bench... ) but it is moving quickly.

comment by seddonm1 · 2021-08-24 · in Apache Arrow Datafusion 5.0.0 release

Are there any good resources on using DataFusion in Python beyond the README [1]? [1] https://github.com/apache/arrow-datafusion/tree/master/pytho...

comment by yadamonk · 2021-08-24 · in Apache Arrow Datafusion 5.0.0 release

…larger than memory datasets. This is implemented in the Ballista crate, which extends DataFusion. https://github.com/apache/arrow-datafusion/tree/master/balli...

comment by andygrove · 2021-08-24 · in Apache Arrow Datafusion 5.0.0 release

OK: I have found: https://github.com/apache/arrow-datafusion "DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.DataFusion supports both an SQL and a DataFrame API for…

comment by pella · 2021-08-24 · in Apache Arrow Datafusion 5.0.0 release

I almost mixed this up with Apache Arrow DataFusion for a second: https://github.com/apache/arrow-datafusion

comment by themaxdavitt · 2021-08-05 · in Datafuse – Modern Real-Time Data Processing in Rust

For sure, I didn't mean to imply they weren't looking at compute too! https://github.com/apache/arrow-datafusion is another example of the shared compute vision. What I was trying to point out is that (at least for Arrow core) they seem to…

comment by BadInformatics · 2021-07-26 · in What's bad about Julia?

…in a break from the mono-repo model. Read more at the announcement[3] or the merge request[4]. [0] https://github.com/apache/arrow-datafusion [1] https://github.com/apache/arrow-rs [2] https://arrow.apache.org/blog/2021/05/04/rust-dev-workf…

comment by michael_j_ward · 2021-05-05 · in Apache Arrow 4.0