Hacker GitHub

← All repos

turboderp/exllama

View on GitHub ↗ ★ 2,946 Python

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

18 mentions on Hacker News by 11 people: 1 story, 17 comments · first 2023-05-23 · latest 2025-03-24

Mentions per year 18 mentions · busiest year 2023 (15)
20082026
AllStoriesComments NewestTopOldest
  1. …a problem? According to this, the GPUs don’t communicate that much once the weights are loaded: https://github.com/turboderp/exllama/discussions/16#discussi... > So at FP16 precision that's a grand total of 16 kB you're transmitting over…
    comment by manmal · 2025-03-24 · in Qwen2.5-VL-32B: Smarter and Lighter
  2. Seems interesting! https://github.com/turboderp/exllama "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights."
    comment by helloericsf · 2024-06-20 · in Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC
  3. …gb/s so 4090 is 10% faster for llama inference than 3090 and more than 2x faster than apple m2 max https://github.com/turboderp/exllama using exllama you can get 160 tokens/s in 7b model and 97 tokens/s in 13b model while m2 max has only…
    comment by instagib · 2024-04-16 · in AMD unveils Ryzen Pro 8000-series processors
  4. …Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2
    comment by aftbit · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  5. one 4090 + 3090-Ti https://github.com/turboderp/exllama
    comment by mromanuk · 2023-08-09 · in Ask HN: Cheapest hardware to run Llama 2 70B
  6. …GPUs, which likely means much better throughput on a single 40GB A100 (or a cheaper 48GB Pro GPU) https://github.com/turboderp/exllama#dual-gpu-results And this is without any consideration of batching (which I am not familiar with TBH).…
    comment by brucethemoose2 · 2023-07-20 · in Llama Is Expensive
  7. ExLlama still uses a bit less VRAM than anything else out there: https://github.com/turboderp/exllama#new-implementation - this is sometimes significant since from my personal experience it can support full context on a quantized llama-33b…
    comment by lhl · 2023-07-19 · in GPT-4 API General Availability
  8. …directly. If you have a high-end Nvidia consumer card (3090/4090) I'd highly recommend looking into https://github.com/turboderp/exllama For those generally confused, the r/LocalLLaMA wiki is a good place to start:…
    comment by lhl · 2023-07-19 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  9. …https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama . If you prefer to use an "instruct" model à la ChatGPT (i.e. that does not need few-shot learning to…
    comment by juliensalinas · 2023-07-11 · in GPT-4 details leaked?
  10. > https://github.com/turboderp/exllama Is exllama an alternative to llama.cpp?
    comment by ignoramous · 2023-07-09 · in GPT-4 API General Availability
  11. …abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/ ) [1] https://github.com/turboderp/exllama#results-so-far [2] https://github.com/aigoopy/llm-jeopardy [3] https://github.com/Troyanovsky/Local-LLM-…
    comment by lhl · 2023-07-07 · in GPT-4 API General Availability
  12. …zippy (faster than AMD's recent live MI300 demo). For llama-based models, recently I've been using https://github.com/turboderp/exllama a lot. It has a Dockerfile/docker-compose.yml so it should be pretty easy to get going. llama.cpp is…
    comment by lhl · 2023-06-18 · in Falcon LLM – A 40B Model
  13. …so there's probably more headroom even) for a relatively easy port. That's really impressive: https://github.com/turboderp/exllama/pull/7 (It's worth noting that for the latter, all you need to do is install the ROCm version of PyTorch and…
    comment by lhl · 2023-06-14 · in AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory
  14. GPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.
    comment by lhl · 2023-06-14 · in AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory
  15. …of reference, currently exllama [1] runs a 4-bit GPTQ of the same 13b model at 83.5 tokens/s. [1] https://github.com/turboderp/exllama
    comment by lhl · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  16. …efficient and really fast [0] https://huggingface.co/docs/transformers/main/model_doc/llam... [1] https://github.com/turboderp/exllama [2] https://github.com/Lightning-AI/lit-llama EDIT: Or do you mean cuda? Because yeah, it's such a shame…
    comment by theaiquestion · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  17. …accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets…
    comment by lhl · 2023-05-23 · in Meta AI announces Massive Multilingual Speech code, models for 1000+ languages