Hacker GitHub

← All repos

spotify/luigi

View on GitHub ↗ ★ 18,782 Python

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

39 mentions on Hacker News by 33 people: 9 stories, 30 comments · first 2012-09-20 · latest 2024-02-27

Mentions per year 39 mentions · busiest year 2018 (9)
20082026
AllStoriesComments NewestTopOldest
  1. …Well, he built https://modal.com , one of the coolest things since sliced mangoes, and before that https://github.com/spotify/luigi
    comment by samuell · 2024-02-27 · in Software Infrastructure 2.0: A Wishlist (2021)
  2. I agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ - https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you…
    comment by idorosen · 2023-09-22 · in Ask HN: What is the correct way to deal with pipelines?
  3. …with a single instance? No database? Are you sure you don't just mean "a scheduler" ala Luigi? https://github.com/spotify/luigi And what kind of scheduler? Again, for "a single instance" it doesn't need to be distributed. For distributed…
    comment by 0xbadcafebee · 2022-05-23 · in Lessons learned from running Apache Airflow at scale
  4. ▲ 4
    Spotify/luigi: build complex pipelines of batch jobs in Python
    story by swyx · 2021-01-15 · 0 comments
  5. Take look at luigi, which is a lightweight task orchestrator with minimalistic dependencies. [1] https://github.com/spotify/luigi
    comment by throwaway7281 · 2021-01-12 · in How to Become a Data Engineer in 2021
  6. …I was able to keep the data pipeline running at that scale without having to rewrite anything. [1] https://github.com/spotify/luigi
    comment by daveFNbuck · 2020-12-31 · in PyPy Project looking for sponsorship to add support for Apple Silicon
  7. Luigi from Spotify: https://github.com/spotify/luigi We’ve been using it for complex update workflows for about 5 yrs now, and it just works. It doesn’t do scheduling or have a fancy ui, but it’s a solid workhorse.
    comment by dfsegoat · 2020-11-24 · in AWS Managed Workflows for Apache Airflow
  8. …- https://github.com/lyft/flyte - https://aws.amazon.com/step-functions/ - https://github.com/spotify/luigi - https://docs.metaflow.org/ - https://github.com/dagster-io/dagster - https://github.com/argoproj/argo - https://github.com/
    comment by knite · 2020-05-30 · in Introduction to Apache Airflow
  9. There's also Luigi. https://github.com/spotify/luigi
    comment by daveFNbuck · 2019-12-26 · in Dolphin Scheduler
  10. Was mario inspired by spotify's luigi? Coincidence? https://github.com/spotify/luigi
    comment by IggleSniggle · 2019-12-05 · in Dgsh – Directed Graph Shell
  11. The following comes to mind Python based: http://www.celeryproject.org/ <-- async task queue https://github.com/spotify/luigi <-- more pipe line centric There are many like it. RQ(more bare bone)
    comment by iAm25626 · 2018-11-27 · in Ask HN: How would you queue and process 10K+ long running jobs
  12. …want to illustrate this and how they don't end up writing their own ORM anyway, and we see, oh https://github.com/spotify/luigi/blob/master/luigi/db_task_h... , it's SQLAlchemy ORM. Foiled again in my search to see this elusive super clean…
    comment by zzzeek · 2018-09-01 · in I don't want to learn your query language
  13. …working with Luigi such a breeze (although we experienced other problems for our use cases). [1] https://github.com/spotify/luigi
    comment by samuell · 2018-08-18 · in Python Fire – Generates CLIs from any Python object
  14. Luigi. We literally use this for anything requiring a series of more than a few steps: https://github.com/spotify/luigi
    comment by dfsegoat · 2018-08-17 · in Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas?
  15. …(especially for a seasoned Python developer) and are used in production environments as well. [0] https://github.com/spotify/luigi [1] https://github.com/apache/incubator-airflow
    comment by danthelion · 2018-08-17 · in Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas?
  16. …was previously developed and maintained by Spotify, as a distributed Make written with Python: https://github.com/spotify/luigi Not sure if Spotify still uses it but it is in their Github org.
    comment by danso · 2018-07-11 · in How ProPublica Illinois Uses GNU Make to Load Data
  17. Check out Luigi (Python -- https://github.com/spotify/luigi ). I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.
    comment by rch · 2018-05-14 · in Ask HN: How do you handle long-running workflows at your company?
  18. …opinionated solution that would have fit my use case perfectly. [1] https://airflow.apache.org/ [2] https://github.com/spotify/luigi [3] The operational complexity of real-time, distributed architectures is non-trivial. You'd be amazed how…
    comment by cosmie · 2018-05-09 · in Mara: A lightweight ETL framework, halfway between plain scripts and Airflow
  19. …DataVersionControl: https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred They all focus in slightly…
    comment by fnl · 2018-01-27 · in Data Science at the Command Line
  20. …workflow tool like Luigi[0] or Airflow[1]. In a phrase, I'd say they're like "Make for data". [0]: https://github.com/spotify/luigi [1]: https://github.com/apache/incubator-airflow
    comment by abd12 · 2017-09-12 · in Ask HN: Data pipelines and ETLs
  21. …come out-of-the-box in the other platforms. So I'm not really sure what makes this unique. [1] https://github.com/spotify/luigi [2] https://airflow.incubator.apache.org/
    comment by nl · 2017-09-06 · in Michelangelo: Uber’s Machine Learning Platform
  22. [Luigi]( https://github.com/spotify/luigi ) out of Spotify sounds exactly like what you're looking for. It allows you to specify dependent tasks, pipe input/output between them, and more.
    comment by vageli · 2017-07-17 · in Ask HN: What tasks do you automate?
  23. Luigi from Spotify is very nice. We use it for computational pipelines in our neuroscience research. https://github.com/spotify/luigi/blob/master/README.rst
    comment by stenl · 2017-06-15 · in Show HN: Your Own Task Queue for Python
  24. ▲ 4
    Ask HN: What do you use to manage chains of long-running jobs?
    story by polm23 · 2017-04-28 · 5 comments
  25. …I like Airflow too, but it did seems a lot more complicated the Luigi when I played with it. [1] https://github.com/spotify/luigi
    comment by nl · 2017-03-01 · in Airflow and the Future of Data Engineering: A Q&A
  26. ▲ 1
    Ask HN: Is your company doing any data quality monitoring or mdm?
    story by cosmie · 2016-12-01 · 0 comments
  27. …IRC/IMAP/FTP/etc. [1] https://github.com/apache/incubator-airflow [2] http://oozie.apache.org/ [3] https://github.com/spotify/luigi [4] https://github.com/cantino/huginn
    comment by reubano · 2016-07-21 · in Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
  28. Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline
    comment by rathboma · 2016-06-06 · in How we run Spark and Sqoop in production
  29. …those ETL jobs? How do you handle ETL dependencies/workflow?(=thinking spotify's luigi[1] here) [1] https://github.com/spotify/luigi
    comment by kakoni · 2016-03-19 · in Engineers Shouldn’t Write ETL
  30. …Cloud Pub/Sub and other parts of our Big Data stack.) Luigi has had BigQuery support for some time: https://github.com/spotify/luigi/pull/1002 Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in…
    comment by crb · 2016-02-23 · in Spotify moves its back end to Google Cloud
  31. ▲ 2
    Spotify Luigi 2.0.0
    story by jcox92 · 2015-10-26 · 0 comments
  32. …whole thing failed or not, easiest way is indeed to have dummy targets. Anyway, take a look at https://github.com/spotify/luigi , it is basically make-like tool geared toward data pipelines. We are experimenting with it for our data…
    comment by oavdeev · 2015-04-02 · in Make: Theory and Practice
  33. yep, check out Spotify's Luigi project. Probably the most widely adopted OSS one https://github.com/spotify/luigi
    comment by unode · 2015-03-12 · in Pinterest open sources Pinball – a flexible data workflow manager
  34. ▲ 1
    Luigi: Python module for building complex pipelines of batch jobs
    story by mpweiher · 2014-12-10 · 0 comments
  35. ▲ 2
    Luigi - Spotify's workflow tool for Hadoop, Command line or any Python workloads
    story by samuell · 2014-08-18 · 0 comments
  36. ▲ 3
    Luigi: Python module for complex pipeline of batch jobs
    story by signa11 · 2013-06-26 · 1 comment
  37. We've built Luigi at Spotify to solve a lot of similar problems: https://github.com/spotify/luigi Might be worth checking out if you are building large data flows. We probably run 10k Luigi "tasks" every day, of which the majority is…
    comment by erikbern · 2013-03-16 · in Introducing Chronos: A Replacement for Cron
  38. ▲ 1
    Spotify open-sourced Luigi, batch job pipeline tool
    story by jyrki · 2012-09-21 · 0 comments
  39. ▲ 2
    Spotify releases "Luigi" - Python module for complex batch jobs
    story by birk · 2012-09-20 · 0 comments