Hacker GitHub

← All repos

rdaum/eider

View on GitHub ↗ ★ 27 Rust

experiment in performant nvfp4 inference on DGX Spark

11 mentions on Hacker News by 1 person: 0 stories, 11 comments · first 2026-07-27 · latest 2026-09-18

Mentions per year 11 mentions · busiest year 2026 (11)
20082026
AllStoriesComments NewestTopOldest
  1. …I'm getting high on my own supply and misunderstand entirely the whole thing, but it seems to work? https://github.com/rdaum/eider/commit/9c2d5c049068c33da2a48b... I don't have the chutzpah to go creating PRs for vLLM to do the same.
    comment by cmrdporcupine · 2026-09-18 · in OpenJev
  2. …-- fork / modify SGLang or vllm or llama.cpp, or do what I did in my bespoke inference engine ( https://github.com/rdaum/eider/ commit https://github.com/rdaum/eider/commit/b2f981b7ebe0e338f60188... ) that ends up being, instead: 1.…
    comment by cmrdporcupine · 2026-09-18 · in OpenJev
  3. …(for DGX Spark, in Rust & CUDA) and get answers pretty much as fast as the Jev openrouter endpoint. https://github.com/rdaum/eider/ It's running over Qwen3.6. Getting it working with Qwen3.8 Flash Next now and getting a battery of tests…
    comment by cmrdporcupine · 2026-09-18 · in OpenJev
  4. …a bunch of (C++) CUDA kernels over, and ground away on them til they got equivalent performance https://github.com/rdaum/eider/tree/main/backends/cuda-oxide And mostly just DeepSeek 4.1 Flash, too. Not even a frontier model. Sorry.
    comment by cmrdporcupine · 2026-09-17 · in Nvidia announces native GPU programming in Rust
  5. …ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing). https://github.com/rdaum/eider/
    comment by cmrdporcupine · 2026-08-26 · in Qwen3.8-Flash-Next
  6. If you've got a DGX Spark try my little engine: https://github.com/rdaum/eider/ NVFP4 quant
    comment by cmrdporcupine · 2026-08-26 · in Qwen3.8-Flash-Next
  7. …inference engine. I've got it (Qwen 3.8 flash next) working (sans ... MTP working on that now). https://github.com/rdaum/eider/ ~80tok/sec prefill, 12tok/sec decode, ~80GiB memory resident, the n-gram table pages from SSD.
    comment by cmrdporcupine · 2026-08-26 · in Qwen3.8-Flash-Next
  8. …with paging (from nVME) of the n-gram table. Resident ~80GiB for weights & context. On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later) Still actively working on…
    comment by cmrdporcupine · 2026-08-26 · in Qwen3.8-Flash-Next
  9. I have this working on a branch of my https://github.com/rdaum/eider (for DGX Spark) nVME paging the n-gram table (in BF16 for now). Still working at it. Prefill sucks still but decode is about 12 tok/sec and the model weights fit nicely…
    comment by cmrdporcupine · 2026-08-26 · in Qwen3.8-Flash-Next
  10. https://github.com/rdaum/eider/ fwiw
    comment by cmrdporcupine · 2026-08-10 · in Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
  11. …that 128GB is going to choke out around 15 tok/sec. Even at NVFP4. My experiments with this are at https://github.com/rdaum/eider/ For my current work I get to log into trays on a real GB300. It's somewhat comical that NVIDIA is marketing…
    comment by cmrdporcupine · 2026-07-27 · in Kimi-K3 on HuggingFace