Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …OK with the extra memory. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e... [2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...
    comment by bakkoting · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  2. Some results here: https://github.com/ggerganov/llama.cpp/discussions/406 tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B.…
    comment by bakkoting · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  3. Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after…
    comment by jart · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  4. …tps://github.com/NouamaneTazi/bloomz.cpp ) compares to the 7B parameter Alpaca model with llama.cpp ( https://github.com/ggerganov/llama.cpp )? I have the latter working on a M1 Macbook Air with very good results for what it is. Curious if…
    comment by JasonZ2 · 2023-03-31 · in Could you train a ChatGPT-beating model for $85k and run it in a browser?
  5. Hi. I wrote the weights file format that llama.cpp uses, as of yesterday https://github.com/ggerganov/llama.cpp/pull/613 What can I do to assist you getting these deltas ready?
    comment by jart · 2023-03-31 · in Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality
  6. llama.cpp is also MIT https://github.com/ggerganov/llama.cpp previously discussed here https://news.ycombinator.com/item?id=35100086 and one of the rust wrapper: https://news.ycombinator.com/item?id=35171527 (also MIT)
    comment by homarp · 2023-03-28 · in Show HN: A fully open-source (Apache 2.0)implementation of llama
  7. …an M1 Macbook Air with 8gig RAM. Gerganov shows the Whisper model running on an Android phone. [0] https://github.com/ggerganov/llama.cpp
    comment by wsgeorge · 2023-03-28 · in ChatGPT works seamlessly with photos and files, enabling numerous new use cases
  8. …trend since the closest nodes of a global delivery network could be several countries away. [0] https://github.com/ggerganov/llama.cpp/discussions/205 [1] https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-...
    comment by wsgeorge · 2023-03-28 · in Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models
  9. …steps you might need to do. If you get stuck, just ask ChatGPT and include any error messages. https://github.com/ggerganov/llama.cpp
    comment by throwaway4aday · 2023-03-26 · in FlexGen: Running large language models on a single GPU
  10. …non-trivial steps in order for everything to be allocated correctly. The changes are here [1]. [1] https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010...
    comment by v64 · 2023-03-18 · in Load LLaMA Models Instantly
  11. …complete } // rest of pre-existing code } You could write a Python extension for this code https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... and see if it works for your use case.
    comment by jart · 2023-03-18 · in Load LLaMA Models Instantly
  12. …the storage space you can download them and use the llama-cpp repo posted here a few days ago ( https://github.com/ggerganov/llama.cpp ) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook…
    comment by NameError · 2023-03-17 · in Load LLaMA Models Instantly
  13. …LLM, you can already run it very easily using llama.cpp and will get the raw output you need. https://github.com/ggerganov/llama.cpp There's already instructions on how anyone can fine-tune it to behave similarly to ChatGPT for as little…
    comment by tarruda · 2023-03-17 · in Stanford Alpaca web demo suspended “until further notice”
  14. …from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...
    comment by adultSwim · 2023-03-17 · in Load LLaMA Models Instantly
  15. If you want to avoid twitter this discusses the changes: https://github.com/ggerganov/llama.cpp/issues/91
    comment by eternalban · 2023-03-17 · in Load LLaMA Models Instantly
  16. …plausible output. It sounds like an objective quality metric would help track down issues. https://github.com/ggerganov/llama.cpp/issues/129 https://github.com/ggerganov/llama.cpp/issues/173
    comment by tveita · 2023-03-17 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  17. There is an interactive mode in llama.cpp https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#... EDIT: I see now you are saying you re-worked the existing interactive mode. I still think your changes could be a PR into the…
    comment by doctoboggan · 2023-03-16 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  18. …and it would be good to have it documented somewhere. There is some discussion about this at https://github.com/ggerganov/llama.cpp/issues/172
    comment by ddren · 2023-03-16 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  19. …implementation only works with CUDA. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/ [2] https://github.com/ggerganov/llama.cpp/issues/9
    comment by comex · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  20. I am working on https://github.com/ggerganov/llama.cpp/issues/77 to "librarify" the code a bit. I eventually did want to Rewrite It In Rust™, but OP just beat me to it.
    comment by tantony · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  21. Here is a thread to tweak the parameters which the model seems very sensitive to: https://github.com/ggerganov/llama.cpp/issues/129
    comment by beiller · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  22. …https://arstechnica.com/information-technology/2023/03/you-c... Here's results of running on Android: https://github.com/ggerganov/llama.cpp/issues/124 This is about running llama on a Raspberry Pi: https://github.com/ggerganov/llama.cpp/is…
    comment by glenneroo · 2023-03-15 · in GPT-4
  23. I feel like https://github.com/ggerganov/llama.cpp/issues/171 is a better approach here? With how fast llama.cpp is changing, this seems like a lot of churn for no reason.
    comment by cozzyd · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  24. All their tensor/math magic seems to happen in https://github.com/ggerganov/llama.cpp/blob/master/ggml.h . So maybe if you implement the ggml.c with tensorflow/libcoral - you'd have a chance.
    comment by saidinesh5 · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  25. …any NPU/TPU patches. By default it only runs on CPU with support for various SIMD extensions. https://github.com/ggerganov/llama.cpp
    comment by saidinesh5 · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  26. That is handy to run llama.cpp https://github.com/ggerganov/llama.cpp https://til.simonwillison.net/llms/llama-7b-m2
    comment by mariuz · 2023-03-15 · in Hetzner launches three new dedicated servers
  27. …takes 14GB of RAM) and then move it to your phone and follow the Android instructions in the repo. https://github.com/ggerganov/llama.cpp/#android I've seen dozens of screenshots of it running in termux on androids by now at completely…
    comment by MacsHeadroom · 2023-03-15 · in Dalai: Automatically install, run, and play with LLaMA on your computer
  28. …my recent fascination with LLMs. This is only going to get wilder as the competition heats up. [0] https://github.com/ggerganov/llama.cpp [1] https://simonwillison.net/2023/Mar/11/llama/
    comment by wsgeorge · 2023-03-14 · in Ask HN: How do you think LLMs will affect society medium/long term?
  29. …prior to running that command? ---------------------------- >> npx dalai llama exec: git clone https://github.com/ggerganov/llama.cpp.git /Users/rickg/llama.cpp in undefined git clone https://github.com/ggerganov/llama.cpp.git…
    comment by evo_9 · 2023-03-14 · in Dalai: Automatically install, run, and play with LLaMA on your computer
  30. …- The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022] https://github.com/ggerganov/llama.cpp/issues/9 - llama.cpp: GPTQ Quantization (3-bit and 4-bit) #9 https://github.com/qwopqwop200/GPTQ-for-LLaMa/ - 4 bits qu
    comment by MacsHeadroom · 2023-03-14 · in Alpaca: A strong open-source instruction-following model
  31. …//github.com/facebookresearch/llama/pull/73 10th March: First commit to llama.cpp by Georgi Gerganov: https://github.com/ggerganov/llama.cpp/commit/26c084662903dd... 11th March: llama.cpp now runs the 7B model on a 4GB RaspberryPi:…
    comment by simonw · 2023-03-13 · in Alpaca: A strong open-source instruction-following model
  32. https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.
    comment by recuter · 2023-03-13 · in How 'Open' Is OpenAI, Really?
  33. See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)
    comment by aseipp · 2023-03-13 · in Using LLaMA with M1 Mac and Python 3.11
  34. It is https://github.com/ggerganov/llama.cpp/blob/master/utils.cpp...
    comment by spion · 2023-03-13 · in Using LLaMA with M1 Mac and Python 3.11
  35. The actual repo's instructions work perfectly without modification under Linux and WSL2: https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  36. How is this post any different than the instructions on the actual repo? https://github.com/ggerganov/llama.cpp
    comment by wjessup · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  37. …magnet link from here https://github.com/shawwn/llama-dl/ To get running, just follow these steps https://github.com/ggerganov/llama.cpp/#usage
    comment by diimdeep · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  38. Source code: https://github.com/ggerganov/llama.cpp
    comment by Const-me · 2023-03-12 · in Large language models are having their Stable Diffusion moment
  39. I can certainly imagine it after seeing https://github.com/ggerganov/llama.cpp Still a couple years out but moving way faster than I would have expected.
    comment by mattnewton · 2023-03-11 · in ChatGPT's API is so good and cheap, it makes most text generating AI obsolete
  40. Thanks to this commit from 8 hours ago: https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e... I have now successfully run the 13B model too! I updated my TIL with details: https://til.simonwillison.net/llms/llama-7b-m2#user-conten…
    comment by simonw · 2023-03-11 · in Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
  41. This commit landed 7 hours ago (since I wrote my TIL): https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
    comment by simonw · 2023-03-11 · in Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
  42. …in 4-bit (and 3-bit) quantization methods and even when compared with uncompressed fp16 inference. https://github.com/ggerganov/llama.cpp/issues/9
    comment by KVFinn · 2023-03-11 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support