Hacker GitHub

← All repos

vllm-project/vllm

View on GitHub ↗ ★ 96,122 Python

A high-throughput and memory-efficient inference and serving engine for LLMs

67 mentions on Hacker News by 51 people: 16 stories, 51 comments · first 2023-06-25 · latest 2026-09-26

Mentions per year 67 mentions · busiest year 2026 (32)
20082026
AllStoriesComments NewestTopOldest
  1. …is the main PR (merged) for using DiffusionGemma (diffusion generally) for Jev-a-likeness in vLLM. https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-26 · in Ollaya – Ollama for open-source, Jev-style decision models
  2. …(well, GLiNER specifically) and diffusion are calling and want their Large Classifier Models back. https://github.com/vllm-project/vllm/pull/57250
    comment by Onavo · 2026-09-22 · in OpenAI is well positioned to fast-follow Jev
  3. …pretty damn close, or even better, results within a week. vLLM has a PR very close to merging: https://github.com/vllm-project/vllm/pull/57250 Kev is an open Jev: https://github.com/jaredpalmer/kev
    comment by verdverm · 2026-09-22 · in OpenAI is well positioned to fast-follow Jev
  4. related, vLLM PR nearing merge readiness for Jev like capabilities with diffusion models https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-22 · in Kev: Tiny Jev-like family of decision models built on top of Qwen3.5
  5. …path means displacing things I use daily :/ I have a watchful eye on the diffusion ~ Jev/Kev PR https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-21 · in MiMo v2.6
  6. …a vLLM PR introducing Jev like capabilities for diffusion models (and more, have not delved deeply) https://github.com/vllm-project/vllm/pull/57250
    comment by verdverm · 2026-09-19 · in I built non-autoregressive decision models with RL a year ago
  7. …are leaving both knowledge and instinctual reasoning on the table with any smaller models, FWIW. https://github.com/vllm-project/vllm/pull/57250
    comment by mmastrac · 2026-09-18 · in OpenJev
  8. There is an open PR for VLLM to do this via DefussionGemma https://github.com/vllm-project/vllm/pull/57250
    comment by mohsen1 · 2026-09-18 · in OpenJev
  9. yup https://github.com/vllm-project/vllm/pull/57250
    comment by mohsen1 · 2026-09-18 · in OpenJev
  10. Sure, just wait for this PR to land : https://github.com/vllm-project/vllm/pull/57250
    comment by user547 · 2026-09-18 · in Jev: The Model That Gives AI the Properties of Code
  11. https://github.com/vllm-project/vllm/pull/57250
    comment by user547 · 2026-09-18 · in Open-sourced jev architecture last year with model,paper and dataset
  12. Any diffusion model is potentially a Jev in disguise: https://github.com/vllm-project/vllm/pull/57250 Runs ~0.2s per decision on my DGX Spark. 10/10 programming language detection 9/10 human language detection 10/12 unit magnitude…
    comment by mmastrac · 2026-09-16 · in Reverse-engineered Jev-like model
  13. …hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624
    comment by joshheitzman · 2026-08-29 · in vLLM v0.28.0
  14. …in 2025 got me to add "#naive" to my notes about vLLM. Total WTF. This should never have been done. https://github.com/vllm-project/vllm/pull/21396#discussion_r...
    comment by yencabulator · 2026-08-27 · in LLMs could control their host machines by exploiting inference engines
  15. …quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme... ).
    comment by throwdbaaway · 2026-08-23 · in Why your local LLM feels dumber than it is
  16. …there was an issue of abysmal performance of models on Strix Halo, that is open for half a year ( https://github.com/vllm-project/vllm/issues/34579#issuecomme... ) and nothing is happening there. They do not care about those use cases.…
    comment by npodbielski · 2026-08-21 · in DiffusionGemma Technical Report
  17. vllm PR for DFlash2: https://github.com/vllm-project/vllm/pull/52816
    comment by verdverm · 2026-08-19 · in DFlash 2: Keep Drafting Parallel
  18. [Feat][Core] Add disk offloading support to SimpleCPUOffloadConnector — #49644 https://github.com/vllm-project/vllm/pull/49644 This adds disk as a tier in the HBM → CPU → Disk KV cache hierarchy. There's also a cluster of related…
    comment by LaurensBER · 2026-08-07 · in DeepSeek V4 Flash 0731
  19. …discord community discussed at the same link the other commenter posted. It is based around this PR https://github.com/vllm-project/vllm/pull/43477
    comment by mtone · 2026-06-16 · in Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?
  20. Seems like a pull request for vLLM was just approved a few minutes ago: https://github.com/vllm-project/vllm/pull/41745 ("Add Gemma4 MTP speculative decoding support")
    comment by joakleaf · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  21. …e.g. vLLM docs list tests they run to ensure that output doesn't change if spec. decoding is used: https://github.com/vllm-project/vllm/blob/main/docs/features... But introducing some threshold to accept other high probability tokens is…
    comment by petu · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  22. https://github.com/vllm-project/vllm/issues/38291
    comment by Klaster_1 · 2026-04-19 · in RotorQuant: Faster Than TurboQuant
  23. …hits of GSM8K performance) and with about 80-100% baseline speed (no improvement, regression): https://github.com/vllm-project/vllm/pull/38479 For those not paying attention, it's probably worth sending this and ongoing discussion for vLLM…
    comment by lhl · 2026-04-19 · in The RAM shortage could last years
  24. …with fp8 quantization enabled (>= 128k context). I run into this error during generation though: https://github.com/vllm-project/vllm/issues/36350 . Looks like an issue with the flash attention backend. But yeah, if you are OK with fp8…
    comment by oakpond · 2026-03-27 · in Intel Announces Arc Pro B70 and Arc Pro B65 GPUs
  25. …tps://github.com/RegularJoe-CEO/vllm/blob/waller-operator-... FlashAttention baseline for comparison: https://github.com/vllm-project/vllm/pull/33860 The kernel achieves ~492-496 TFLOPS consistently across all sequence lengths. Looking for…
    comment by luxiedge · 2026-02-05 · in Constant 14ms attention: 512→524K tokens (24.5x faster than FlashAttention)
  26. …image_height] Also, if you’re running the model with vLLM > 0.11.0, you might be hitting this bug: https://github.com/vllm-project/vllm/issues/29595
    comment by spherelot · 2025-12-03 · in Qwen3-VL can scan two-hour videos and pinpoint nearly every detail
  27. …- verl for RL algorithms: https://github.com/volcengine/verl - vLLM for efficient rollouts: https://github.com/vllm-project/vllm - AgentOps for collecting training data (telemetry): https://github.com/AgentOps-AI/agentops - LangGraph for…
    comment by ultmaster · 2025-08-11 · in Tune self-correct SQL agent with RL: AgentLightning+verl+vLLM+AgentOps+LangGraph
  28. …the model so you can't re-create it from scratch. [0] https://github.com/google-deepmind/gemma [1] https://github.com/vllm-project/vllm/pull/2964
    comment by zackangelo · 2025-06-26 · in Introducing Gemma 3n
  29. …to say, "Don't change it. It works." -- [1]: https://hub.docker.com/r/vllm/vllm-openai/tags [2]: https://github.com/vllm-project/vllm/issues/13306 [3]: These kinds of workarounds tend to end up accumulating and never get reviewed back: -…
    comment by tt726259 · 2025-06-23 · in Nano-Vllm: Lightweight vLLM implementation built from scratch
  30. No, vLLM is a thing for serving language models: https://github.com/vllm-project/vllm
    comment by IanCal · 2025-06-23 · in Nano-Vllm: Lightweight vLLM implementation built from scratch
  31. …5 v0.8.3 vLLM release - https://blog.vllm.ai/2025/04/05/llama4.html , so before the QKNorm fix https://github.com/vllm-project/vllm/pull/16311 ) - it seemed pretty decent to me. For English, on a combination of MixEval, LiveBench, IFEval,…
    comment by lhl · 2025-04-24 · in Llama 4 Smells Bad
  32. …found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311 If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for…
    comment by danielhanchen · 2025-04-24 · in Llama 4 Smells Bad
  33. …and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856
    comment by Zambyte · 2025-04-20 · in Gemma 3 QAT Models: Bringing AI to Consumer GPUs
  34. …Deepseek models as of 3 weeks ago. 3x higher generation throughput and 10x token memory capacity. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 MHA is still faster in low QPS regime apparently.…
    comment by refibrillator · 2025-02-24 · in DeepSeek Open Source FlashMLA – MLA Decoding Kernel for Hopper GPUs
  35. Yes, I've used the v3.2 3B-Instruct model in a Slack app. Specifically using vLLM, with a template: https://github.com/vllm-project/vllm/blob/main/examples/tool... Works as expected if you provide a few system prompts with context.
    comment by scriptsmith · 2024-10-24 · in Quantized Llama models with increased speed and a reduced memory footprint
  36. …We're working on expanding our model support. Hence (almost) any model. Thanks for testing! :) https://github.com/vllm-project/vllm/issues/6479 - Billy :)
    comment by billycao · 2024-07-24 · in Show HN: We made glhf.chat – run almost any open-source LLM, including 405B
  37. …[4] https://github.com/bitsandbytes-foundation/bitsandbytes [5] https://github.com/vllm-project/vllm [6] https://ai.meta.com/blog/llama-2/ [7] https://www.nomic.ai/gpt4all [8] http://ollama.ai/ [9] https://github.com/turbo
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  38. …difficulties in vllm which significantly affect the determinism of any model run on it: https://github.com/vllm-project/vllm/issues/966 Note that a suggested fix is upcasting to float32. So it's possible that GPT-3.5 is using an especially…
    comment by nicklecompte · 2024-04-30 · in AI leaderboards are no longer useful. It's time to switch to Pareto curves
  39. The easiest is to use vllm ( https://github.com/vllm-project/vllm ) to run it on a Couple of A100's, and you can benchmark this using this library ( https://github.com/EleutherAI/lm-evaluation-harness )
    comment by varunvummadi · 2024-04-10 · in Mistral AI Launches New 8x22B MOE Model
  40. …or Llama.cpp[1]. Seems to be Llama.cpp via 'Nitro', which was discussed here before [2]. [0] https://github.com/vllm-project/vllm [1] https://github.com/ggerganov/llama.cpp [2] https://news.ycombinator.com/item?id=38887531
    comment by lagrange77 · 2024-03-21 · in Jan: An open source alternative to ChatGPT that runs on the desktop
  41. …and you run multiple inference sessions over the same loaded weights). Obviously projects like https://github.com/vllm-project/vllm exist but we needed to build out a scheduler that can run a fleet of GPUs for a matrix of text/image vs…
    comment by binocarlos · 2024-02-08 · in How we got fine-tuning Mistral-7B to not suck
  42. FYI, vLLM also just added experimental multi-lora support: https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Also check out the new prefix caching, I see huge potential for batch processing purposes there!
    comment by semmulder · 2024-01-31 · in Mistral CEO confirms 'leak' of new open source AI model nearing GPT4 performance
  43. …8x7B: https://mistral.ai/news/mixtral-of-experts/ Official PR into vLLM shows the inference code: https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6... New HuggingFace explainer on MoE very nice: https://huggingface.co/blog/moe…
    comment by BryanLegend · 2023-12-11 · in Mixtral of experts
  44. …My model implements the HF chat_templates feature, and I worked on adding support for that to vLLM: https://github.com/vllm-project/vllm/pull/1756 (llama.cpp is talking about adding support for it too) so I could easily swap out my model…
    comment by Tostino · 2023-12-02 · in OpenAI delays launch of custom GPT store until early 2024
  45. …implement it properly) will work seamlessly with a clone of the OpenAI chat/completions endpoint. https://github.com/vllm-project/vllm/pull/1756 Now that I have that out of the way, back to model training ;).
    comment by Tostino · 2023-12-01 · in Are Open-Source Large Language Models Catching Up?
  46. Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune…
    comment by Tostino · 2023-10-19 · in Optimizing LLM Latency
  47. …per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them?…
    comment by kkielhofner · 2023-10-07 · in AMD may get across the CUDA moat
  48. …does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2
    comment by aftbit · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  49. source code: https://github.com/vllm-project/vllm
    comment by notpublic · 2023-09-14 · in Efficient Memory Management for Large Language Model Serving with PagedAttention
  50. We're finding that when running Llama-2-7B with vLLM ( https://github.com/vllm-project/vllm ) on an A40 GPU we're getting consistently lower time-to-first-token and lower average token generation time than GPT-3.5, even when processing…
    comment by kcorbitt · 2023-09-12 · in Fine-tune your own Llama 2 to replace GPT-3.5/4