Hacker GitHub

← All repos

InternLM/lmdeploy

View on GitHub ↗ ★ 8,140 Python

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

3 mentions on Hacker News by 1 person: 0 stories, 3 comments · first 2023-08-25 · latest 2023-10-07

Mentions per year 3 mentions · busiest year 2023 (3)
20082026
AllStoriesComments NewestTopOldest
  1. I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 You know what’s missing from…
    comment by kkielhofner · 2023-10-07 · in AMD may get across the CUDA moat
  2. vLLM has healthy competition. Not affiliated but try lmdeploy: https://github.com/InternLM/lmdeploy In my testing it’s significantly faster and more memory efficient than vLLM when configured with AWQ int4 and int8 KV cache. If you look at…
    comment by kkielhofner · 2023-09-08 · in NVIDIA introduces TensorRT-LLM for accelerating LLM inference on H100/A100 GPUs
  3. …1): 105 tokens/s This is out of the box, I haven't spent any time further optimizing it. [0] - https://github.com/InternLM/lmdeploy [1] - https://github.com/InternLM/lmdeploy/blob/main/docs/en/kv_in... [2] -…
    comment by kkielhofner · 2023-08-25 · in Code Llama, a state-of-the-art large language model for coding