Hacker GitHub

← All repos

triton-inference-server/tensorrtllm_backend

View on GitHub ↗ ★ 944

The Triton TensorRT-LLM Backend

3 mentions on Hacker News by 1 person: 0 stories, 3 comments · first 2024-01-08 · latest 2024-05-25

Mentions per year 3 mentions · busiest year 2024 (3)
20082026
AllStoriesComments NewestTopOldest
  1. TensorRT-LLM with Triton Inference Server is the fastest in Nvidia land. https://github.com/triton-inference-server/tensorrtllm_backe...
    comment by kkielhofner · 2024-05-25 · in How fast can one reasonably expect to get inference on a ~70B model?
  2. Nvidia Triton Inference Server with the TensorRT-LLM backend: https://github.com/triton-inference-server/tensorrtllm_backe... It’s used by Mistral, AWS, Cloudflare, and countless others. vLLM, HF TGI, Rayserve, etc are certainly viable but…
    comment by kkielhofner · 2024-02-09 · in OpenAI compatibility
  3. I support any progress to erode the Nvidia monopoly. That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia…
    comment by kkielhofner · 2024-01-08 · in MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference