intentee/paddler
View on GitHub ↗ ★ 1,692 Rust
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
4 mentions on Hacker News by 4 people: 2 stories, 2 comments · first 2024-05-18 · latest 2024-07-16
20082026
-
Just got https://github.com/distantmagic/paddler working across 2 machines on windows, for load balancing, This will be next level and useful for Llama 400B to run across multiple machines. But looks like windows support is not there yet.
comment by Jayakumark · 2024-07-16 · in Exo: Run your own AI cluster at home with everyday devices -
Looks like it’s to only have one completion running at a time currently, https://github.com/distantmagic/paddler/blob/main/loadbalanc... I was curious if there was any other cache tracking happening
comment by tjfontaine · 2024-06-02 · in Show HN: Open-source load balancer for llama.cpp -
▲ 151
Show HN: Open-source load balancer for llama.cppstory by mcharytoniuk · 2024-06-01 · 20 comments
-
▲ 3
Paddler – Stateful load balancer custom-tailored for llama.cppstory by throwaway888abc · 2024-05-18 · 0 comments