intentee/paddler

View on GitHub ↗ ★ 1,692 Rust

Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.

4 mentions on Hacker News by 4 people: 2 stories, 2 comments · first 2024-05-18 · latest 2024-07-16

Mentions per year 4 mentions · busiest year 2024 (4)

20082026

All Stories Comments Newest Top Oldest

▲ 151
Show HN: Open-source load balancer for llama.cpp
story by mcharytoniuk · 2024-06-01 · 20 comments
▲ 3
Paddler – Stateful load balancer custom-tailored for llama.cpp
story by throwaway888abc · 2024-05-18 · 0 comments