Hacker GitHub

← All repos

vllm-project/vllm

View on GitHub ↗ ★ 96,122 Python

A high-throughput and memory-efficient inference and serving engine for LLMs

67 mentions on Hacker News by 51 people: 16 stories, 51 comments · first 2023-06-25 · latest 2026-09-26

Mentions per year 67 mentions · busiest year 2026 (32)
20082026
AllStoriesComments NewestTopOldest
  1. …is the main PR (merged) for using DiffusionGemma (diffusion generally) for Jev-a-likeness in vLLM. https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-26 · in Ollaya – Ollama for open-source, Jev-style decision models
  2. …(well, GLiNER specifically) and diffusion are calling and want their Large Classifier Models back. https://github.com/vllm-project/vllm/pull/57250
    comment by Onavo · 2026-09-22 · in OpenAI is well positioned to fast-follow Jev
  3. …pretty damn close, or even better, results within a week. vLLM has a PR very close to merging: https://github.com/vllm-project/vllm/pull/57250 Kev is an open Jev: https://github.com/jaredpalmer/kev
    comment by verdverm · 2026-09-22 · in OpenAI is well positioned to fast-follow Jev
  4. related, vLLM PR nearing merge readiness for Jev like capabilities with diffusion models https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-22 · in Kev: Tiny Jev-like family of decision models built on top of Qwen3.5
  5. …path means displacing things I use daily :/ I have a watchful eye on the diffusion ~ Jev/Kev PR https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-21 · in MiMo v2.6
  6. …a vLLM PR introducing Jev like capabilities for diffusion models (and more, have not delved deeply) https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-19 · in I built non-autoregressive decision models with RL a year ago
  7. …are leaving both knowledge and instinctual reasoning on the table with any smaller models, FWIW. https://github.com/vllm-project/vllm/pull/57250
    comment by mmastrac · 2026-09-18 · in OpenJev
  8. There is an open PR for VLLM to do this via DefussionGemma https://github.com/vllm-project/vllm/pull/57250
    comment by mohsen1 · 2026-09-18 · in OpenJev
  9. yup https://github.com/vllm-project/vllm/pull/57250
    comment by mohsen1 · 2026-09-18 · in OpenJev
  10. Sure, just wait for this PR to land : https://github.com/vllm-project/vllm/pull/57250
    comment by user547 · 2026-09-18 · in Jev: The Model That Gives AI the Properties of Code
  11. https://github.com/vllm-project/vllm/pull/57250
    comment by user547 · 2026-09-18 · in Open-sourced jev architecture last year with model,paper and dataset
  12. Any diffusion model is potentially a Jev in disguise: https://github.com/vllm-project/vllm/pull/57250 Runs ~0.2s per decision on my DGX Spark. 10/10 programming language detection 9/10 human language detection 10/12 unit magnitude…
    comment by mmastrac · 2026-09-16 · in Reverse-engineered Jev-like model
  13. ▲ 1
    vLLM: Jev-like mode for the DiffusionGemma model
    story by mmastrac · 2026-09-16 · 0 comments
  14. ▲ 3
    vLLM 0.29.0: Model Runner V2 is now the default
    story by joshcsimmons · 2026-09-09 · 0 comments
  15. …hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624
    comment by joshheitzman · 2026-08-29 · in vLLM v0.28.0
  16. ▲ 108
    vLLM v0.28.0
    story by mrrrcs · 2026-08-29 · 43 comments
  17. …in 2025 got me to add "#naive" to my notes about vLLM. Total WTF. This should never have been done. https://github.com/vllm-project/vllm/pull/21396#discussion_r...
    comment by yencabulator · 2026-08-27 · in LLMs could control their host machines by exploiting inference engines
  18. …quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme... ).
    comment by throwdbaaway · 2026-08-23 · in Why your local LLM feels dumber than it is
  19. …there was an issue of abysmal performance of models on Strix Halo, that is open for half a year ( https://github.com/vllm-project/vllm/issues/34579#issuecomme... ) and nothing is happening there. They do not care about those use cases.…
    comment by npodbielski · 2026-08-21 · in DiffusionGemma Technical Report
  20. vllm PR for DFlash2: https://github.com/vllm-project/vllm/pull/52816
    comment by verdverm · 2026-08-19 · in DFlash 2: Keep Drafting Parallel
  21. [Feat][Core] Add disk offloading support to SimpleCPUOffloadConnector — #49644 https://github.com/vllm-project/vllm/pull/49644 This adds disk as a tier in the HBM → CPU → Disk KV cache hierarchy. There's also a cluster of related…
    comment by LaurensBER · 2026-08-07 · in DeepSeek V4 Flash 0731
  22. …discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477
    comment by mtone · 2026-06-16 · in Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?
  23. Seems like a pull request for vLLM was just approved a few minutes ago: https://github.com/vllm-project/vllm/pull/41745 ("Add Gemma4 MTP speculative decoding support")
    comment by joakleaf · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  24. …e.g. vLLM docs list tests they run to ensure that output doesn't change if spec. decoding is used: https://github.com/vllm-project/vllm/blob/main/docs/features... But introducing some threshold to accept other high probability tokens is…
    comment by petu · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  25. https://github.com/vllm-project/vllm/issues/38291
    comment by Klaster_1 · 2026-04-19 · in RotorQuant: Faster Than TurboQuant
  26. …hits of GSM8K performance) and with about 80-100% baseline speed (no improvement, regression): https://github.com/vllm-project/vllm/pull/38479 For those not paying attention, it's probably worth sending this and ongoing discussion for vLLM…
    comment by lhl · 2026-04-19 · in The RAM shortage could last years
  27. ▲ 4
    vLLM IR: A Functional Intermediate Representation for vLLM
    story by matt_d · 2026-04-07 · 0 comments
  28. ▲ 5
    vLLM introduces memory optimizations for long-context inference
    story by addisud · 2026-04-04 · 0 comments
  29. …with fp8 quantization enabled (>= 128k context). I run into this error during generation though: https://github.com/vllm-project/vllm/issues/36350 . Looks like an issue with the flash attention backend. But yeah, if you are OK with fp8…
    comment by oakpond · 2026-03-27 · in Intel Announces Arc Pro B70 and Arc Pro B65 GPUs
  30. ▲ 2
    vLLM (high-throughput LLM serving engine)
    story by roody_wurlitzer · 2026-02-25 · 0 comments
  31. …tps://github.com/RegularJoe-CEO/vllm/blob/waller-operator-... FlashAttention baseline for comparison: https://github.com/vllm-project/vllm/pull/33860 The kernel achieves ~492-496 TFLOPS consistently across all sequence lengths. Looking for…
    comment by luxiedge · 2026-02-05 · in Constant 14ms attention: 512→524K tokens (24.5x faster than FlashAttention)
  32. ▲ 1
    vLLM multi-turn conversations design
    story by CCs · 2026-01-22 · 0 comments
  33. …image_height] Also, if you’re running the model with vLLM > 0.11.0, you might be hitting this bug: https://github.com/vllm-project/vllm/issues/29595
    comment by spherelot · 2025-12-03 · in Qwen3-VL can scan two-hour videos and pinpoint nearly every detail
  34. …- verl for RL algorithms: https://github.com/volcengine/verl - vLLM for efficient rollouts: https://github.com/vllm-project/vllm - AgentOps for collecting training data (telemetry): https://github.com/AgentOps-AI/agentops - LangGraph for…
    comment by ultmaster · 2025-08-11 · in Tune self-correct SQL agent with RL: AgentLightning+verl+vLLM+AgentOps+LangGraph
  35. …the model so you can't re-create it from scratch. [0] https://github.com/google-deepmind/gemma [1] https://github.com/vllm-project/vllm/pull/2964
    comment by zackangelo · 2025-06-26 · in Introducing Gemma 3n
  36. …to say, "Don't change it. It works." -- [1]: https://hub.docker.com/r/vllm/vllm-openai/tags [2]: https://github.com/vllm-project/vllm/issues/13306 [3]: These kinds of workarounds tend to end up accumulating and never get reviewed back: -…
    comment by tt726259 · 2025-06-23 · in Nano-Vllm: Lightweight vLLM implementation built from scratch
  37. No, vLLM is a thing for serving language models: https://github.com/vllm-project/vllm
    comment by IanCal · 2025-06-23 · in Nano-Vllm: Lightweight vLLM implementation built from scratch
  38. …5 v0.8.3 vLLM release - https://blog.vllm.ai/2025/04/05/llama4.html , so before the QKNorm fix https://github.com/vllm-project/vllm/pull/16311 ) - it seemed pretty decent to me. For English, on a combination of MixEval, LiveBench, IFEval,…
    comment by lhl · 2025-04-24 · in Llama 4 Smells Bad
  39. …found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311 If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for…
    comment by danielhanchen · 2025-04-24 · in Llama 4 Smells Bad
  40. …and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856
    comment by Zambyte · 2025-04-20 · in Gemma 3 QAT Models: Bringing AI to Consumer GPUs
  41. …Deepseek models as of 3 weeks ago. 3x higher generation throughput and 10x token memory capacity. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 MHA is still faster in low QPS regime apparently.…
    comment by refibrillator · 2025-02-24 · in DeepSeek Open Source FlashMLA – MLA Decoding Kernel for Hopper GPUs
  42. Yes, I've used the v3.2 3B-Instruct model in a Slack app. Specifically using vLLM, with a template: https://github.com/vllm-project/vllm/blob/main/examples/tool... Works as expected if you provide a few system prompts with context.
    comment by scriptsmith · 2024-10-24 · in Quantized Llama models with increased speed and a reduced memory footprint
  43. …We're working on expanding our model support. Hence (almost) any model. Thanks for testing! :) https://github.com/vllm-project/vllm/issues/6479 - Billy :)
    comment by billycao · 2024-07-24 · in Show HN: We made glhf.chat – run almost any open-source LLM, including 405B
  44. …[4] https://github.com/bitsandbytes-foundation/bitsandbytes [5] https://github.com/vllm-project/vllm [6] https://ai.meta.com/blog/llama-2/ [7] https://www.nomic.ai/gpt4all [8] http://ollama.ai/ [9] https://github.com/turbo
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  45. …difficulties in vllm which significantly affect the determinism of any model run on it: https://github.com/vllm-project/vllm/issues/966 Note that a suggested fix is upcasting to float32. So it's possible that GPT-3.5 is using an especially…
    comment by nicklecompte · 2024-04-30 · in AI leaderboards are no longer useful. It's time to switch to Pareto curves
  46. The easiest is to use vllm ( https://github.com/vllm-project/vllm ) to run it on a Couple of A100's, and you can benchmark this using this library ( https://github.com/EleutherAI/lm-evaluation-harness )
    comment by varunvummadi · 2024-04-10 · in Mistral AI Launches New 8x22B MOE Model
  47. …or Llama.cpp[1]. Seems to be Llama.cpp via 'Nitro', which was discussed here before [2]. [0] https://github.com/vllm-project/vllm [1] https://github.com/ggerganov/llama.cpp [2] https://news.ycombinator.com/item?id=38887531
    comment by lagrange77 · 2024-03-21 · in Jan: An open source alternative to ChatGPT that runs on the desktop
  48. …and you run multiple inference sessions over the same loaded weights). Obviously projects like https://github.com/vllm-project/vllm exist but we needed to build out a scheduler that can run a fleet of GPUs for a matrix of text/image vs…
    comment by binocarlos · 2024-02-08 · in How we got fine-tuning Mistral-7B to not suck
  49. FYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
    comment by semmulder · 2024-01-31 · in Mistral CEO confirms 'leak' of new open source AI model nearing GPT4 performance
  50. ▲ 1
    VLLM Sacrifices Accuracy for Speed
    story by behnamoh · 2024-01-24 · 0 comments