turbolytics/sql-flow on Hacker News

…data: - Configuration Driven - CI Test framework integrated - First class metrics and monitoring https://github.com/turbolytics/sql-flow I wrote about some of these while launching SQLFlow on the render platform. I'm happy to answer any…

comment by dm03514 · 2026-10-08 · in Data needs its DevOps moment

Thank you, fixed: https://github.com/turbolytics/sql-flow/commit/e7cc28b7bdd2f... docs are updated: https://turbolytics.io/demos/bluesky

comment by dm03514 · 2026-09-17 · in Show HN: Real-Time Bluesky Post Firehose in < 256MB

…I built on top of DuckDB. Looking forward to what duckdb provides in terms of perf out of the box! https://github.com/turbolytics/sql-flow DuckDB has been a fantastic engine to build on (in python), and processes thousands of events per…

comment by dm03514 · 2026-08-17 · in A Preview of DuckDB v2.0

…if you could create an issue in the repo describing your use case and desired functionality a bit! https://github.com/turbolytics/sql-flow/issues We were looking at solving some of the simplier use cases first before branching out into…

comment by dm03514 · 2025-12-08 · in Show HN: DuckDB for Kafka Stream Processing

Still hacking on some data tools: DuckDB for stream processing: https://github.com/turbolytics/sql-flow Lightweight kafka stream processing using DuckDB as the execution engine. 300MiB runtime can easily handle thousands of messages /…

comment by dm03514 · 2025-11-10 · in Ask HN: What Are You Working On? (Nov 2025)

…around duckdb that allows for high performance stream processing and a rich connector ecosystem: https://github.com/turbolytics/sql-flow Building a company around a tool is hard. There's been some interest but streaming is kind of…

comment by dm03514 · 2025-07-27 · in Ask HN: What are you working on? (July 2025)

Built a stream processing engine using duckdb https://github.com/turbolytics/sql-flow It has some interest, unfortunately building tools as a business strategy is rough. Beginning to work on first actual product! More soon :)

comment by dm03514 · 2025-06-30 · in Ask HN: What Are You Working On? (June 2025)

Duckdb for streaming data! https://github.com/turbolytics/sql-flow I think the industry lacks lightweight fully featured stream processing solutions. I think it’s either heavyweight jvm or bespoke custom solutions Sqlflow aims to be a…

comment by dm03514 · 2025-04-28 · in Ask HN: What are you working on? (April 2025)

…provide over sql? I have a side project called Sqlflow which is attempting to do something similar/ https://github.com/turbolytics/sql-flow It’s not a DSL but the pipeline is standardized using the source, process, sink stages. Right now…

comment by dm03514 · 2025-04-14 · in Ask HN: Why don't we have a functional DSL for data+embedding+API pipelines?

…high performance dev-centric streaming solution powered by duckdb, it’s getting a little traction https://github.com/turbolytics/sql-flow Iceberg has been one of the most popular feature so far: https://sql-flow.com/docs/tutorials/iceberg-s…

comment by dm03514 · 2025-04-10 · in Arroyo (YC W23) has been acquired by Cloudflare

…format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it! https://github.com/turbolytics/sql-flow Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For…

comment by dm03514 · 2025-03-18 · in Preview: Amazon S3 Tables and Lakehouse in DuckDB

…to write streaming data to s3 in iceberg format: https://sql-flow.com/docs/tutorials/iceberg-sink https://github.com/turbolytics/sql-flow I realize it's not quite what you asked for but wanted to mention it. I'm surprised at lack of native…

comment by dm03514 · 2025-03-18 · in Preview: Amazon S3 Tables and Lakehouse in DuckDB

I'm building a stream processing framework using DuckDB. https://github.com/turbolytics/sql-flow The goal is to create a stream processing framework that supports SQL jobs. Apache Flink supports this but is very heavy-weight overall. I…

comment by dm03514 · 2025-02-24 · in Ask HN: What are you working on? (February 2025)

We just added Clickhouse sink support to SQLFlow: - https://github.com/turbolytics/sql-flow/issues/100 - https://github.com/turbolytics/sql-flow/pull/116 The Clickhouse python API made this trivial thanks to supporting Arrow! We have a…

comment by dm03514 · 2025-02-15 · in Bulk inserts on ClickHouse: How to avoid overstuffing your instance

Hello! Just released a version of SQLFlow that adds support for streaming data to iceberg tables. https://github.com/turbolytics/sql-flow?tab=readme-ov-file#s... pyiceberg does the heavy lifting. It is a great project with a lot of active…

comment by dm03514 · 2025-01-15 · in Show HN: Execute SQL against Bluesky firehose

…I just added support for websocket sources which enables sql over the Bluesky firehouse. https://github.com/turbolytics/sql-flow?tab=readme-ov-file#c... Duckdb does all the sql execution, and python is responsible for sourcing the data.…

comment by dm03514 · 2024-12-31 · in Show HN: Execute SQL against Bluesky firehose

…The new in-memory processor easily handles 45,000 messages / second for simple aggregation. https://github.com/turbolytics/sql-flow?tab=readme-ov-file#b... ------ I'd really appreciate feedback on features that would make this more useful.…

comment by dm03514 · 2024-03-29 · in SQLFlow: DuckDB for Stream Processing – 45k messages/second