apache/hamilton
View on GitHub ↗ ★ 2,622 Jupyter Notebook
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
19 mentions on Hacker News by 4 people: 5 stories, 14 comments · first 2023-03-07 · latest 2024-12-18
20082026
-
…sitting in the same subnet, which isn’t the definition of a “distributed task” I’m going for. [1] https://github.com/DAGWorks-Inc/hamilton
comment by halfcat · 2024-12-18 · in Show HN: I built an open-source data pipeline tool in Go -
Yep. When I was designing https://github.com/dagworks-inc/hamilton part of the idea was to make it easy to understand what and where. That is, enable one to grep for function definitions and their downstream use easily, and where people…
comment by krawczstef · 2024-09-04 · in Greppability is an underrated code metric -
Heh, this was very much the design philosophy behind Hamilton (github.com/dagworks-inc/hamilton). The basic idea was that if you have a data artifact (columns for dataframes initially), you should be able to ctrl-f and find it in your…
comment by elijahbenizzy · 2024-09-03 · in Greppability is an underrated code metric -
…as state machines https://github.com/dagworks-inc/burr - Hamilton -- build dataflows as DAGs: https://github.com/dagworks-inc/hamilton Looking for feedback -- we had some good initial traction on HN, and are looking for OS…
comment by elijahbenizzy · 2024-08-28 · in Ask HN: What are you working on (August 2024)? -
▲ 40
Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelinesstory by elijahbenizzy · 2024-05-02 · 10 comments
-
▲ 94
Show HN: Burr – A framework for building and debugging GenAI apps fasterstory by elijahbenizzy · 2024-04-03 · 22 comments
-
…This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton ).
comment by krawczstef · 2024-03-08 · in Show HN: Hatchet – Open-source distributed task queue -
…with it -- https://blog.dagworks.io/p/building-a-lightweight-experiment . You can see the code here https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/... . Usually we've been prototyping with streamlit, but found that at times to…
comment by krawczstef · 2024-03-02 · in FastUI: Build Better UIs Faster -
▲ 2
Show HN: On Garbage Collection and Memory Optimization in Hamiltonstory by elijahbenizzy · 2023-10-24 · 1 comment
-
…it for another spin? [disclaimer I'm a maintainer of Hamilton] Otherwise FYI Prophet gels well with https://github.com/DAGWorks-Inc/hamilton for setting up your features and dataset for fitting & prediction[/disclaimer].
comment by krawczstef · 2023-09-26 · in Prophet: Automatic Forecasting Procedure -
▲ 4
Show HN: Declarative Spark Transformations with Hamiltonstory by elijahbenizzy · 2023-08-24 · 0 comments
-
Hey folks, co-creator of Hamilton (github.com/dagworks-inc/hamilton) here. We just released a new set of features enabling dynamism in Hamilton DAGs, and this blog post demonstrates it by counting unique stargazers in github across the…
comment by elijahbenizzy · 2023-08-15 · in Using dynamic DAGs in Hamilton to count GitHub stars -
…than "it depends". To that end - we've come up with a post showing how one might use Hamilton ( https://github.com/dagWorks-Inc/hamilton ) to easily create a workflow to ingest data into a vector database that I think has a great…
comment by krawczstef · 2023-07-11 · in Langchain Is Pointless -
Totally! As a person driving a project like https://github.com/DAGWorks-Inc/hamilton I couldn't agree more!
comment by krawczstef · 2023-07-09 · in Langchain Is Pointless -
From a nuts and bolts perspective, I've been thinking of building some reactivity on top of https://github.com/dagworks-inc/hamilton (author here) that could get at this. (If you have a use case that could be documented, I'd appreciate it.)
comment by krawczstef · 2023-05-10 · in IPyflow: Reactive Python Notebooks in Jupyter(Lab) -
Yeah! So we actually have an integration with polars. See https://github.com/DAGWorks-Inc/hamilton/blob/5c8e564d19ff23... . To be clear, the specific paradigm we're referring to is this way of writing transforms as functions where the…
comment by elijahbenizzy · 2023-03-07 · in Launch HN: DAGWorks – ML platform for data science teams -
…[2]: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws... [3]: https://github.com/DAGWorks-Inc/hamilton/pull/105
comment by data_ders · 2023-03-07 · in Launch HN: DAGWorks – ML platform for data science teams -
…it's a little trickier but doable -- we were just going over this with a user recently actually! https://github.com/DAGWorks-Inc/hamilton/issues/90
comment by elijahbenizzy · 2023-03-07 · in Launch HN: DAGWorks – ML platform for data science teams -
▲ 182
Launch HN: DAGWorks – ML platform for data science teamsstory by krawczstef · 2023-03-07 · 65 comments