Neroued/ninfer
View on GitHub ↗ ★ 3,233 C++
High-performance single-GPU inference for selected model checkpoints and GPUs.
12 mentions on Hacker News by 10 people: 3 stories, 9 comments · first 2026-07-23 · latest 2026-10-04
20082026
-
This is interesting, thanks. - https://github.com/Neroued/ninfer
comment by snehesht · 2026-10-04 · in Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s -
Just a couple days ago I learned about ninfer ( https://github.com/Neroued/ninfer ) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.
comment by pllbnk · 2026-09-03 · in Qwen 3.8 27B available on Cerebras at 1500 tokens/s -
try ninfer once you get your 5090 https://github.com/Neroued/ninfer
comment by vardalab · 2026-08-31 · in Apple caught off guard by AI demand for Mac Mini and Mac Studio -
…Gives really only one concurrent session that flies because kv caching is not perfect for ninfer https://github.com/Neroued/ninfer both are very serviceable, I prefer FP8 on 2xR9700 But, yes it doesn't scale that well but in 5 years the…
comment by vardalab · 2026-08-31 · in Apple caught off guard by AI demand for Mac Mini and Mac Studio -
Strongly recommend https://github.com/Neroued/ninfer , which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B.
comment by Philpax · 2026-08-25 · in Qwen 3.8-Flash-Next releasing tomorrow (125B a6B) -
…a 5090, and the two 3090s in my server were cheap enough when amortised over several years. [0]: https://github.com/Neroued/ninfer
comment by Philpax · 2026-08-21 · in Building an (almost) fully self-hosted, sandboxed, agentic software factory -
…with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures. https://github.com/Neroued/ninfer/blob/master/docs/performan... Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code…
comment by Foobar8568 · 2026-08-17 · in Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU -
If you've got RTX 5090, maybe try ninfer ( https://github.com/Neroued/ninfer ). Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer (NVFP4; 256k ctx).
comment by iv42 · 2026-08-17 · in Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU -
If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.
comment by kimsey0 · 2026-08-14 · in Qwen 3.8 27B