Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …steps you might need to do. If you get stuck, just ask ChatGPT and include any error messages. https://github.com/ggerganov/llama.cpp
    comment by throwaway4aday · 2023-03-26 · in FlexGen: Running large language models on a single GPU
  2. …non-trivial steps in order for everything to be allocated correctly. The changes are here [1]. [1] https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010...
    comment by v64 · 2023-03-18 · in Load LLaMA Models Instantly
  3. …complete } // rest of pre-existing code } You could write a Python extension for this code https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... and see if it works for your use case.
    comment by jart · 2023-03-18 · in Load LLaMA Models Instantly
  4. …the storage space you can download them and use the llama-cpp repo posted here a few days ago ( https://github.com/ggerganov/llama.cpp ) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook…
    comment by NameError · 2023-03-17 · in Load LLaMA Models Instantly
  5. …LLM, you can already run it very easily using llama.cpp and will get the raw output you need. https://github.com/ggerganov/llama.cpp There's already instructions on how anyone can fine-tune it to behave similarly to ChatGPT for as little…
    comment by tarruda · 2023-03-17 · in Stanford Alpaca web demo suspended “until further notice”
  6. …from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...
    comment by adultSwim · 2023-03-17 · in Load LLaMA Models Instantly
  7. If you want to avoid twitter this discusses the changes: https://github.com/ggerganov/llama.cpp/issues/91
    comment by eternalban · 2023-03-17 · in Load LLaMA Models Instantly
  8. ▲ 2
    Inference at the Edge
    story by georgehill · 2023-03-17 · 0 comments
  9. …plausible output. It sounds like an objective quality metric would help track down issues. https://github.com/ggerganov/llama.cpp/issues/129 https://github.com/ggerganov/llama.cpp/issues/173
    comment by tveita · 2023-03-17 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  10. ▲ 4
    Inference at the Edge
    story by Mizza · 2023-03-16 · 0 comments
  11. There is an interactive mode in llama.cpp https://github.com/ggerganov/llama.cpp/blob/master/main.cpp#... EDIT: I see now you are saying you re-worked the existing interactive mode. I still think your changes could be a PR into the…
    comment by doctoboggan · 2023-03-16 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  12. …and it would be good to have it documented somewhere. There is some discussion about this at https://github.com/ggerganov/llama.cpp/issues/172
    comment by ddren · 2023-03-16 · in Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook
  13. …implementation only works with CUDA. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/ [2] https://github.com/ggerganov/llama.cpp/issues/9
    comment by comex · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  14. I am working on https://github.com/ggerganov/llama.cpp/issues/77 to "librarify" the code a bit. I eventually did want to Rewrite It In Rust™, but OP just beat me to it.
    comment by tantony · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  15. Here is a thread to tweak the parameters which the model seems very sensitive to: https://github.com/ggerganov/llama.cpp/issues/129
    comment by beiller · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  16. …https://arstechnica.com/information-technology/2023/03/you-c... Here's results of running on Android: https://github.com/ggerganov/llama.cpp/issues/124 This is about running llama on a Raspberry Pi: https://github.com/ggerganov/llama.cpp/is…
    comment by glenneroo · 2023-03-15 · in GPT-4
  17. I feel like https://github.com/ggerganov/llama.cpp/issues/171 is a better approach here? With how fast llama.cpp is changing, this seems like a lot of churn for no reason.
    comment by cozzyd · 2023-03-15 · in Llama.rs – Rust port of llama.cpp for fast LLaMA inference on CPU
  18. All their tensor/math magic seems to happen in https://github.com/ggerganov/llama.cpp/blob/master/ggml.h . So maybe if you implement the ggml.c with tensorflow/libcoral - you'd have a chance.
    comment by saidinesh5 · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  19. …any NPU/TPU patches. By default it only runs on CPU with support for various SIMD extensions. https://github.com/ggerganov/llama.cpp
    comment by saidinesh5 · 2023-03-15 · in LLaMa running at 5 tokens/second on a Pixel 6
  20. That is handy to run llama.cpp https://github.com/ggerganov/llama.cpp https://til.simonwillison.net/llms/llama-7b-m2
    comment by mariuz · 2023-03-15 · in Hetzner launches three new dedicated servers
  21. …takes 14GB of RAM) and then move it to your phone and follow the Android instructions in the repo. https://github.com/ggerganov/llama.cpp/#android I've seen dozens of screenshots of it running in termux on androids by now at completely…
    comment by MacsHeadroom · 2023-03-15 · in Dalai: Automatically install, run, and play with LLaMA on your computer
  22. …my recent fascination with LLMs. This is only going to get wilder as the competition heats up. [0] https://github.com/ggerganov/llama.cpp [1] https://simonwillison.net/2023/Mar/11/llama/
    comment by wsgeorge · 2023-03-14 · in Ask HN: How do you think LLMs will affect society medium/long term?
  23. …prior to running that command? ---------------------------- >> npx dalai llama exec: git clone https://github.com/ggerganov/llama.cpp.git /Users/rickg/llama.cpp in undefined git clone https://github.com/ggerganov/llama.cpp.git…
    comment by evo_9 · 2023-03-14 · in Dalai: Automatically install, run, and play with LLaMA on your computer
  24. …- The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022] https://github.com/ggerganov/llama.cpp/issues/9 - llama.cpp: GPTQ Quantization (3-bit and 4-bit) #9 https://github.com/qwopqwop200/GPTQ-for-LLaMa/ - 4 bits qu
    comment by MacsHeadroom · 2023-03-14 · in Alpaca: A strong open-source instruction-following model
  25. …//github.com/facebookresearch/llama/pull/73 10th March: First commit to llama.cpp by Georgi Gerganov: https://github.com/ggerganov/llama.cpp/commit/26c084662903dd... 11th March: llama.cpp now runs the 7B model on a 4GB RaspberryPi:…
    comment by simonw · 2023-03-13 · in Alpaca: A strong open-source instruction-following model
  26. https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.
    comment by recuter · 2023-03-13 · in How 'Open' Is OpenAI, Really?
  27. See https://github.com/ggerganov/llama.cpp/issues/62 (the related repo was originally posted on 4chan, is all, but the code is on GitHub)
    comment by aseipp · 2023-03-13 · in Using LLaMA with M1 Mac and Python 3.11
  28. It is https://github.com/ggerganov/llama.cpp/blob/master/utils.cpp...
    comment by spion · 2023-03-13 · in Using LLaMA with M1 Mac and Python 3.11
  29. ▲ 6
    LLaMA 7B model running on 4GB RAM Raspberry Pi 4
    story by amrrs · 2023-03-12 · 0 comments
  30. The actual repo's instructions work perfectly without modification under Linux and WSL2: https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  31. How is this post any different than the instructions on the actual repo? https://github.com/ggerganov/llama.cpp
    comment by wjessup · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  32. …magnet link from here https://github.com/shawwn/llama-dl/ To get running, just follow these steps https://github.com/ggerganov/llama.cpp/#usage
    comment by diimdeep · 2023-03-12 · in Using LLaMA with M1 Mac and Python 3.11
  33. Source code: https://github.com/ggerganov/llama.cpp
    comment by Const-me · 2023-03-12 · in Large language models are having their Stable Diffusion moment
  34. I can certainly imagine it after seeing https://github.com/ggerganov/llama.cpp Still a couple years out but moving way faster than I would have expected.
    comment by mattnewton · 2023-03-11 · in ChatGPT's API is so good and cheap, it makes most text generating AI obsolete
  35. Thanks to this commit from 8 hours ago: https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e... I have now successfully run the 13B model too! I updated my TIL with details: https://til.simonwillison.net/llms/llama-7b-m2#user-conten…
    comment by simonw · 2023-03-11 · in Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
  36. This commit landed 7 hours ago (since I wrote my TIL): https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
    comment by simonw · 2023-03-11 · in Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
  37. …in 4-bit (and 3-bit) quantization methods and even when compared with uncompressed fp16 inference. https://github.com/ggerganov/llama.cpp/issues/9
    comment by KVFinn · 2023-03-11 · in Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
  38. ▲ 989
    Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
    story by mrtksn · 2023-03-10 · 284 comments