Hacker GitHub

← All repos

kvcache-ai/ktransformers

View on GitHub ↗ ★ 19,651 Python

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

14 mentions on Hacker News by 11 people: 2 stories, 12 comments · first 2024-08-29 · latest 2026-03-24

Mentions per year 14 mentions · busiest year 2025 (10)
20082026
AllStoriesComments NewestTopOldest
  1. …produce such a histogram, though currently its activation statistics are just a ".pt" file. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... FWIW I never got it to work and did not dig into it much.
    comment by Gracana · 2026-03-24 · in Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
  2. …and hold the remaining weights in DDR5 ram. Can see their deployment guide for reference here: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
    comment by oceansweep · 2025-11-07 · in Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model
  3. State of the art of local models is even further. For example, look into https://github.com/kvcache-ai/ktransformers , which achieve >11 tokens/s on a relatively old two socket Xeon servers + retail RTX 4090 GPU. Even more interesting is…
    comment by p12tic · 2025-06-01 · in Why DeepSeek is cheap at scale but expensive to run locally
  4. This is the state of the art for such a setup. Really good performance! https://github.com/kvcache-ai/ktransformers
    comment by phonon · 2025-05-28 · in Deepseek R1-0528
  5. Take a look at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
    comment by phonon · 2025-04-05 · in The Llama 4 herd
  6. …not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU side https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... The final step of going to 6 experts instead of 8 feels like cheating (not a…
    comment by zamadatix · 2025-03-06 · in DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
  7. …) 2. AMD Epyc + 4090 or 5090 running KTransformers (over 10 t/s decode ? https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... )
    comment by wolfgangK · 2025-03-06 · in Apple M3 Ultra
  8. …They tried both single and dual CPU, with the latter about 30% faster....not necessarily worth it. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
    comment by phonon · 2025-02-20 · in Grok3 Launch [video]
  9. …split the model with MOEs running on CPUs and MLAs running on GPU. See the ktransformers project: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... This takes advantage of the sparsity of MOE and the efficient KV-cache of MLA.
    comment by fspeech · 2025-02-18 · in Grok3 Launch [video]
  10. …and be smart about which layers go where. This project seems to be doing that with great results: * https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
    comment by Eisenstein · 2025-02-11 · in Building a personal, private AI computer on a budget
  11. …the GPU for context processing. If your GPU has enough VRAM to support it, you might benefit from https://github.com/kvcache-ai/ktransformers
    comment by ConnectX-2 · 2024-11-17 · in Tencent Hunyuan-Large
  12. Using https://github.com/kvcache-ai/ktransformers/ , an intel/amd laptop with 128GB RAM and 16GB VRAM can run the IQ4_XS quant and decode about 4-7 token/s, depending on RAM speed and context size. Using llama.cpp, the decoding speed is…
    comment by throwdbaaway · 2024-10-31 · in DeepSeek v2.5 – open-source LLM comparable to GPT-4, but 95% less expensive