local-inference-lab/rtx6kpro
View on GitHub ↗ ★ 1,197 Python
RTX 6000 Pro Wiki — Running Large LLMs (Qwen3.5-397B, Kimi-K2.5, GLM-5) on PCIe GPUs without NVLink
5 mentions on Hacker News by 3 people: 0 stories, 5 comments · first 2026-05-10 · latest 2026-08-14
20082026
-
…things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... . Disregard references to RTX 6000 cards, most of it is generally applicable…
comment by CamperBob2 · 2026-08-14 · in Qwen 3.8 27B -
You're gonna want a custom vLLM build. Here's a runbook: https://github.com/local-inference-lab/rtx6kpro/blob/master/... If the newer builds aren't working, you might try running the old v6 build (based on the eldritch-enlightenment…
comment by lcampbell · 2026-08-07 · in DeepSeek V4 Flash 0731 -
Not OP, but I am seeing up to 260 tokens/second output at c=1 with the recipe at https://github.com/local-inference-lab/rtx6kpro/blob/master/... using 4x 6k cards. Average is more like 200. There may be a way to get the 2-bit quantized…
comment by CamperBob2 · 2026-06-16 · in Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding? -
…like Deepseek 4 Flash, Minimax 2.7 etc.) Join the RTX6kPRO tribe! - https://discord.gg/pYCvaQTf - https://github.com/local-inference-lab/rtx6kpro
comment by biddit · 2026-05-22 · in Was my $48K GPU server worth it? -
…one time. Spring for 8 RTX6000s instead of 4, and you can use the full-precision K2.6 weights ( https://github.com/local-inference-lab/rtx6kpro/blob/master/... ).
comment by CamperBob2 · 2026-05-10 · in Local AI needs to be the norm