Hacker GitHub

← All repos

NVIDIA/TensorRT-LLM

View on GitHub ↗ ★ 14,991 Python

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

12 mentions on Hacker News by 10 people: 3 stories, 9 comments · first 2023-10-20 · latest 2026-04-19

Mentions per year 12 mentions · busiest year 2024 (8)
20082026
AllStoriesComments NewestTopOldest
  1. …was so interesting, because it allows for much higher throughput for a given latency budget: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source... . Serverless would make more sense if we had a significant underutilization…
    comment by rushingcreek · 2024-09-06 · in Phind-405B and faster, high quality AI answers for everyone
  2. …https://www.nomic.ai/gpt4all [8] http://ollama.ai/ [9] https://github.com/turboderp/exllamav2 [10] https://github.com/NVIDIA/TensorRT-LLM [11] https://mistral.ai/news/mixtral-of-experts/ [12] https://cohere.com/blog/command-r-plus-microsoft…
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  3. …rebuild and might be able to be reconfigured to work: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/TensorRT-LLM
    comment by compressedgas · 2024-05-10 · in Are there any alternatives to Nvidia's AI Chat with RTX?
  4. …https://github.com/ml-explore/mlx [2] https://en.wikipedia.org/wiki/Torch_(machine_learning) [3] https://github.com/NVIDIA/TensorRT-LLM
    comment by jmorgan · 2024-04-29 · in Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
  5. https://github.com/NVIDIA/TensorRT-LLM
    comment by alecco · 2024-02-14 · in Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC
  6. …then wrapper around the following github repos: https://github.com/NVIDIA/trt-llm-rag-windows https://github.com/NVIDIA/TensorRT-LLM It's quite a thin wrapper around putting both projects into %LocalAppData%, along with a miniconda…
    comment by operator-name · 2024-02-13 · in Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC
  7. Yeah, seems a bit odd because the TensorRT-LLM repo lists Turing as supported architecture. https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file#pr...
    comment by operator-name · 2024-02-13 · in Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC
  8. …TensorRT-LLM/Triton but of course those comparison benchmarks won't help sell their startup. [0] - https://github.com/NVIDIA/TensorRT-LLM [1] - https://github.com/triton-inference-server/tensorrtllm_backe... [2] -…
    comment by kkielhofner · 2024-01-08 · in MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference
  9. Performance: https://github.com/NVIDIA/TensorRT-LLM/blob/release/0.5.0/do...
    comment by hkab · 2023-10-20 · in Nvidia Releases TensorRT-LLM