Hacker GitHub

← All repos

turboderp-org/exllamav2

View on GitHub ↗ ★ 4,648 Python

A fast inference library for running LLMs locally on modern consumer-class GPUs

9 mentions on Hacker News by 8 people: 1 story, 8 comments · first 2023-09-13 · latest 2024-07-16

Mentions per year 9 mentions · busiest year 2023 (6)
20082026
AllStoriesComments NewestTopOldest
  1. …[6] https://ai.meta.com/blog/llama-2/ [7] https://www.nomic.ai/gpt4all [8] http://ollama.ai/ [9] https://github.com/turboderp/exllamav2 [10] https://github.com/NVIDIA/TensorRT-LLM [11] https://mistral.ai/news/mixtral-of-experts/ [12]…
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  2. …rtx 3090ti runs llama2-7b at 179 tps also using around 5GB 1000GB/s/5gb = 200 https://github.com/turboderp/exllamav2
    comment by slimesli · 2024-04-21 · in Running Llama3 Locally
  3. …Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275 And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...
    comment by cosmojg · 2024-04-08 · in Mixture-of-Depths: Dynamically allocating compute in transformers
  4. …of modest. ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100: https://github.com/turboderp/exllamav2#performance I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my…
    comment by brucethemoose2 · 2023-10-31 · in Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
  5. …or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2
    comment by aftbit · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  6. …Quantization like ExLlama's EX2 or even llama.cpp's k-quants is already far from brute force: https://github.com/turboderp/exllamav2#exl2-quantization > The format allows for mixing quantization levels within a model to achieve any average…
    comment by brucethemoose2 · 2023-09-24 · in LLM Falcon 180B Needs 720GB RAM to Run
  7. …are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it. [0] https://github.com/turboderp/exllamav2#exl2-quantization [1] https://arxiv.org/abs/2210.17323
    comment by mijoharas · 2023-09-13 · in Exllamav2: Inference library for running LLMs locally on consumer-class GPUs
  8. …are used for different layers or weights. Here are more details about the quantization scheme: https://github.com/turboderp/exllamav2#exl2-quantization
    comment by ftxbro · 2023-09-13 · in Exllamav2: Inference library for running LLMs locally on consumer-class GPUs