Hacker GitHub

← All repos

lyogavin/airllm

View on GitHub ↗ ★ 41,574 Jupyter Notebook

AirLLM 2.8T LLM inference with single 4GB GPU

9 mentions on Hacker News by 9 people: 4 stories, 5 comments · first 2024-08-01 · latest 2026-08-07

Mentions per year 9 mentions · busiest year 2026 (6)
20082026
AllStoriesComments NewestTopOldest
  1. For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0
    comment by imenani · 2026-08-03 · in AirLLM 70B inference with single 4GB GPU
  2. …of SSDs will be significantly slower than RAM. That said, there are folks out there doing it. https://github.com/lyogavin/airllm is one example.
    comment by timschmidt · 2026-01-28 · in Trinity large: An open 400B sparse MoE model
  3. I have not used it but this is probably it: https://github.com/lyogavin/airllm
    comment by skirmish · 2025-09-12 · in Qwen3-Next
  4. …models. You can run the biggest models such as llama 3.1 405B on a fairly low end desktop today: https://github.com/lyogavin/airllm However, it will be far slower as you said.
    comment by ryao · 2024-11-20 · in Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference
  5. https://github.com/lyogavin/airllm What sort of speed do you get?
    comment by langcss · 2024-08-01 · in Show HN: Run Llama3.1 405B on a 8GB VRAM challenge [video]