Hacker GitHub

← All repos

vllm-project/vllm

View on GitHub ↗ ★ 96,122 Python

A high-throughput and memory-efficient inference and serving engine for LLMs

67 mentions on Hacker News by 51 people: 16 stories, 51 comments · first 2023-06-25 · latest 2026-09-26

Mentions per year 67 mentions · busiest year 2026 (32)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 2
    Easy, fast, and cheap LLM serving for everyone
    story by vincent_s · 2023-12-17 · 0 comments
  2. ▲ 1
    vllm
    story by tosh · 2023-12-15 · 0 comments
  3. ▲ 1
    Mixtral Expert Parallelism
    story by tosh · 2023-12-15 · 0 comments
  4. …8x7B: https://mistral.ai/news/mixtral-of-experts/ Official PR into vLLM shows the inference code: https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6... New HuggingFace explainer on MoE very nice: https://huggingface.co/blog/moe…
    comment by BryanLegend · 2023-12-11 · in Mixtral of experts
  5. ▲ 2
    Official PR Reveals the Inference Code for Mixtral 8x7B
    story by georgehill · 2023-12-11 · 0 comments
  6. …My model implements the HF chat_templates feature, and I worked on adding support for that to vLLM: https://github.com/vllm-project/vllm/pull/1756 (llama.cpp is talking about adding support for it too) so I could easily swap out my model…
    comment by Tostino · 2023-12-02 · in OpenAI delays launch of custom GPT store until early 2024
  7. …implement it properly) will work seamlessly with a clone of the OpenAI chat/completions endpoint. https://github.com/vllm-project/vllm/pull/1756 Now that I have that out of the way, back to model training ;).
    comment by Tostino · 2023-12-01 · in Are Open-Source Large Language Models Catching Up?
  8. Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune…
    comment by Tostino · 2023-10-19 · in Optimizing LLM Latency
  9. …per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them?…
    comment by kkielhofner · 2023-10-07 · in AMD may get across the CUDA moat
  10. …does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2
    comment by aftbit · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  11. source code: https://github.com/vllm-project/vllm
    comment by notpublic · 2023-09-14 · in Efficient Memory Management for Large Language Model Serving with PagedAttention
  12. We're finding that when running Llama-2-7B with vLLM ( https://github.com/vllm-project/vllm ) on an A40 GPU we're getting consistently lower time-to-first-token and lower average token generation time than GPT-3.5, even when processing…
    comment by kcorbitt · 2023-09-12 · in Fine-tune your own Llama 2 to replace GPT-3.5/4
  13. ▲ 3
    Vllm: High-throughput and memory-efficient inference and serving engine for LLMs
    story by tosh · 2023-09-10 · 0 comments
  14. …of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite…
    comment by kkielhofner · 2023-09-09 · in NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs
  15. ▲ 15
    Ask HN: Are you training and running custom LLMs and how are you doing it?
    story by kordlessagain · 2023-08-14 · 2 comments
  16. ▲ 3
    Vllm
    story by kordlessagain · 2023-08-06 · 0 comments
  17. ▲ 2
    VLLM
    story by sherlockxu · 2023-06-25 · 0 comments