kvcache-ai/ktransformers
View on GitHub ↗ ★ 19,651 Python
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
14 mentions on Hacker News by 11 people: 2 stories, 12 comments · first 2024-08-29 · latest 2026-03-24
20082026
-
…produce such a histogram, though currently its activation statistics are just a ".pt" file. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... FWIW I never got it to work and did not dig into it much.
comment by Gracana · 2026-03-24 · in Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon -
…and hold the remaining weights in DDR5 ram. Can see their deployment guide for reference here: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
comment by oceansweep · 2025-11-07 · in Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model -
State of the art of local models is even further. For example, look into https://github.com/kvcache-ai/ktransformers , which achieve >11 tokens/s on a relatively old two socket Xeon servers + retail RTX 4090 GPU. Even more interesting is…
comment by p12tic · 2025-06-01 · in Why DeepSeek is cheap at scale but expensive to run locally -
This is the state of the art for such a setup. Really good performance! https://github.com/kvcache-ai/ktransformers
comment by phonon · 2025-05-28 · in Deepseek R1-0528 -
Take a look at https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
comment by phonon · 2025-04-05 · in The Llama 4 herd -
…not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU side https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... The final step of going to 6 experts instead of 8 feels like cheating (not a…
comment by zamadatix · 2025-03-06 · in DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon -
…) 2. AMD Epyc + 4090 or 5090 running KTransformers (over 10 t/s decode ? https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... )
comment by wolfgangK · 2025-03-06 · in Apple M3 Ultra -
…They tried both single and dual CPU, with the latter about 30% faster....not necessarily worth it. https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
comment by phonon · 2025-02-20 · in Grok3 Launch [video] -
…split the model with MOEs running on CPUs and MLAs running on GPU. See the ktransformers project: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... This takes advantage of the sparsity of MOE and the efficient KV-cache of MLA.
comment by fspeech · 2025-02-18 · in Grok3 Launch [video] -
…and be smart about which layers go where. This project seems to be doing that with great results: * https://github.com/kvcache-ai/ktransformers/blob/main/doc/en...
comment by Eisenstein · 2025-02-11 · in Building a personal, private AI computer on a budget -
▲ 14
Show HN: KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefillstory by sssummer · 2025-02-10 · 0 comments
-
…the GPU for context processing. If your GPU has enough VRAM to support it, you might benefit from https://github.com/kvcache-ai/ktransformers
comment by ConnectX-2 · 2024-11-17 · in Tencent Hunyuan-Large -
Using https://github.com/kvcache-ai/ktransformers/ , an intel/amd laptop with 128GB RAM and 16GB VRAM can run the IQ4_XS quant and decode about 4-7 token/s, depending on RAM speed and context size. Using llama.cpp, the decoding speed is…
comment by throwdbaaway · 2024-10-31 · in DeepSeek v2.5 – open-source LLM comparable to GPT-4, but 95% less expensive -
▲ 20
Show HN: KTransformers–236B Model and 1M Context LLM Inference on Local Machinesstory by sssummer · 2024-08-29 · 3 comments