Hacker GitHub

← All repos

qwopqwop200/GPTQ-for-LLaMa

View on GitHub ↗ ★ 3,069 Python

4 bits quantization of LLaMA using GPTQ

20 mentions on Hacker News by 15 people: 0 stories, 20 comments · first 2023-03-10 · latest 2023-07-11

Mentions per year 20 mentions · busiest year 2023 (20)
20082026
AllStoriesComments NewestTopOldest
  1. …might need to apply 4-bit quantization with something like https://github.com/PanQiWei/AutoGPTQ or https://github.com/qwopqwop200/GPTQ-for-LLaMa . Then you can improve the inference speed by using https://github.com/turboderp/exllama . If…
    comment by juliensalinas · 2023-07-11 · in GPT-4 details leaked?
  2. With a single NVIDIA 3090 and the fastest inference branch of GPTQ-for-LLAMA https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/fastest-i... , I get a healthy 10-15 tokens per second on the 30B models. IMO GGML is great (And I totally use…
    comment by oceanplexian · 2023-06-06 · in GGML – AI at the Edge
  3. https://github.com/qwopqwop200/GPTQ-for-LLaMa
    comment by sp332 · 2023-05-25 · in QLoRA: Efficient Finetuning of Quantized LLMs
  4. …models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwop200/GPTQ-for-LLaMa And performing the quantization step is memory and time expensive. Which is why some kind…
    comment by andy_xor_andrew · 2023-05-25 · in QLoRA: Efficient Finetuning of Quantized LLMs
  5. …can just specify a longer size on the CLI for llama.cpp if you are OK with the extra memory. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e... [2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...
    comment by bakkoting · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  6. …is .3 on 7B model, which is not insignificant. ggml quirks are definitely need to be fixed. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa
    comment by terafo · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  7. …with right group size. If you'd like to know more, I think you should read GPTQ paper[2]. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa [2] https://arxiv.org/abs/2210.17323
    comment by terafo · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  8. …- https://github.com/hizkifw/WebChatRWKVstic - https://github.com/ggerganov/whisper.cpp - https://github.com/qwopqwop200/GPTQ-for-LLaMa - https://github.com/oobabooga/text-generation-webui/issues/14... -…
    comment by lxe · 2023-03-22 · in Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
  9. quantization takes lots of RAM: https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/main/READ... says llama-13B takes 42GB and 33B takes more than 64GB...
    comment by homarp · 2023-03-20 · in The genie escapes: Stanford copies the ChatGPT AI for less than $600
  10. Pretty much. Llama specific: https://github.com/qwopqwop200/GPTQ-for-LLaMa > According to GPTQ paper, As the size of the model increases, the difference in performance between FP16 and GPTQ decreases. https://nolanoorg.substack.com/p/int-4-…
    comment by recuter · 2023-03-17 · in TextSynth Server
  11. There are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on 32GB machines, 33B and lower would run on 16GB machines, and 13B (minimal useful one seems like) would fit on 8GB -- which…
    comment by recuter · 2023-03-16 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  12. …support it yet [2] though it will soon. And the original implementation only works with CUDA. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/ [2] https://github.com/ggerganov/llama.cpp/issues/9
    comment by comex · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  13. …implementation[1] ran some tests using the same quantization algorithm as llama.cpp (4 bit RTN). 1: https://github.com/qwopqwop200/GPTQ-for-LLaMa
    comment by ddren · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  14. Some numbers here: https://github.com/qwopqwop200/GPTQ-for-LLaMa#result
    comment by summarity · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  15. …https://github.com/ggerganov/llama.cpp/issues/9 - llama.cpp: GPTQ Quantization (3-bit and 4-bit) #9 https://github.com/qwopqwop200/GPTQ-for-LLaMa/ - 4 bits quantization of LLaMa using GPTQ
    comment by MacsHeadroom · 2023-03-14 · in Alpaca: A strong open-source instruction-following model
  16. …after quantization, so initial benchmarks aren't relevant. Here are some benchmarks for quantized https://github.com/qwopqwop200/GPTQ-for-LLaMa
    comment by diimdeep · 2023-03-12 · in ChatGPT's API is so good and cheap, it makes most text generating AI obsolete
  17. …fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss. https://github.com/qwopqwop200/GPTQ-for-LLaMa
    comment by MacsHeadroom · 2023-03-11 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
  18. …GPTQ LLaMA models are the current top-performers. This site has done a lot of the heavy lifting: https://github.com/qwopqwop200/GPTQ-for-LLaMa With 30b-4bit on a RTX 4090, I'm seeing numbers like: Output generated in 4.17 seconds (4.03…
    comment by 0xbadc0de5 · 2023-03-10 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
  19. …will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention). [0] https://github.com/qwopqwop200/GPTQ-for-LLaMa [1] https://arxiv.org/abs/2210.17323 [2] https://github.com/oobabooga/text-generation-webui/issue…
    comment by MacsHeadroom · 2023-03-10 · in LLaMA Text-Generation-Webui
  20. I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa , and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very…
    comment by lxe · 2023-03-10 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support