← All repos

tensorninja/ninfer-4090

View on GitHub ↗ ★ 16 C++

From-scratch C++/CUDA engine for Qwen3.8-27B on one RTX 4090: 3,600 tok/s prefill, 149 tok/s MTP decode, full 262K context. Works with any coding agent over OpenAI and Anthropic APIs, best with Caudra. Per-request LoRA adapters, plus OpenAI Decisions and TypeSafe System One on the same weights.

4 mentions on Hacker News by 1 person: 0 stories, 4 comments · first 2026-08-26 · latest 2026-11-05