sgl-project/sglang
View on GitHub ↗ ★ 38,367 Python
SGLang is a high-performance serving framework for large language models and multimodal models.
14 mentions on Hacker News by 13 people: 1 story, 13 comments · first 2024-02-24 · latest 2026-10-15
20082026
-
▲ 1
SGLang – High-performance serving framework for LLM's and multimodal modelsstory by peter_d_sherman · 2026-10-15 · 0 comments
-
…and I'd rather it were a floor, so concurrency adapts at runtime without tuning. Upstream PR: https://github.com/sgl-project/sglang/pull/34623 I'm happy to go into scheduler details. I'd especially like to hear from anyone who has handled…
comment by apejcic · 2026-09-26 · in Prefill Concurrency in SGLang: Consistent TTFT Under Multi-Tenant Load -
…5, DeepSeek V4. Only now is it slowly strating to get optimized in the major inference engines: ( https://github.com/sgl-project/sglang/issues/19380 https://github.com/sgl-project/sglang/pull/22851 etc.). Of course, DS V4 adds extra…
comment by Palmik · 2026-05-23 · in DeepSeek makes the V4 Pro price discount permanent -
…the required newlines from their previous format). You can see their tool call parsing code here: https://github.com/sgl-project/sglang/blob/34013d9d5a591e3c0...
comment by reissbaker · 2025-12-22 · in GLM-4.7: Advancing the Coding Capability -
…I have a ton of respect for SGLang as a runtime. I'm hoping something can be done there. https://github.com/sgl-project/sglang/discussions/4461 . As noted in that thread, it is really great that Qwen3-Coder has a tool-parser built-in:…
comment by jauntywundrkind · 2025-08-08 · in I want everything local – Building my offline AI workspace -
…And here are the instructions to replicate the particular benchmark: https://github.com/sgl-project/sglang/issues/7227 The LLM text I linked in my original answer carries out the math using the energy consumption of the NVidia
comment by pama · 2025-07-25 · in Distillation makes AI models smaller and cheaper -
…prefix and decoding serving (eg https://docs.vllm.ai/examples/online_serving/disaggregated_s... or https://github.com/sgl-project/sglang/issues/3554 and https://github.com/sgl-project/sglang/issues/4655 ) — could your solution enable a…
comment by pama · 2025-06-28 · in Lossless LLM 3x Throughput Increase by LMCache -
…is achievable today. In the mean time Intel guys are working on something even more impressive. In https://github.com/sgl-project/sglang/pull/5150 they claim that they achieve >15 tokens/s generation and >350 tokens/s prefill. They don't…
comment by p12tic · 2025-06-01 · in Why DeepSeek is cheap at scale but expensive to run locally -
8x H200s recommended: https://github.com/sgl-project/sglang/tree/main/benchmark/de...
comment by bavell · 2024-12-27 · in DeepSeek-V3 -
…multiple valid tokens at once, rather than just one token at a time. Corresponding project link: https://github.com/sgl-project/sglang
comment by OutOfHere · 2024-10-19 · in .txt raises $11.9M to make language models programmable -
…will have nearly twice as reported speeds with INT8 (fp16 A100 benched here https://github.com/sgl-project/sglang?tab=readme-ov-file#ben... )
comment by iAkashPaul · 2024-08-24 · in Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands -
…using two approaches to get this running faster: - structured generation techniques from sglang ( https://github.com/sgl-project/sglang ) that allow them to generate faster JSON (with look-ahead / pre-fill) with strong guarantees on the…
comment by fzysingularity · 2024-03-19 · in Inversion: Fast, Reliable Structured LLMs -
…0. https://github.com/outlines-dev/outlines 1. https://github.com/guidance-ai/guidance 2. https://github.com/sgl-project/sglang
comment by PheonixPharts · 2024-03-12 · in Show HN: Prompts as WASM Programs -
…https://github.com/search?q=repo%3Apredibase%2Florax+rocm&ty... https://github.com/sgl-project/sglang/issues/157 https://github.com/casper-hansen/AutoAWQ (supports rocm)
comment by latchkey · 2024-02-24 · in AMD ROCm Software Blogs