vllm-project/vllm
View on GitHub ↗ ★ 96,122 Python
A high-throughput and memory-efficient inference and serving engine for LLMs
67 mentions on Hacker News by 51 people: 16 stories, 51 comments · first 2023-06-25 · latest 2026-09-26
20082026
-
…of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite…
comment by kkielhofner · 2023-09-09 · in NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs