Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. I think llama.cpp might be easier to set up and get running. https://github.com/ggerganov/llama.cpp
    comment by jbandela1 · 2023-05-03 · in OpenLLaMA: An Open Reproduction of LLaMA
  2. There's some info here: https://github.com/ggerganov/llama.cpp/issues/1240
    comment by ac29 · 2023-04-30 · in Whisper.cpp v1.4.0
  3. ▲ 97
    llama.cpp: Roadmap May 2023
    story by tosh · 2023-04-30 · 6 comments
  4. …build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp
    comment by PostOnce · 2023-04-30 · in A brief history of LLaMA models
  5. …of the models out in the open really have GPT-3-ish capabilities, not even llama-65b. That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable…
    comment by int_19h · 2023-04-28 · in What's up in the Python community? – April 2023
  6. …because these models are constrained by memory bandwidth and not compute (low arithmetic density) https://github.com/ggerganov/llama.cpp
    comment by remorses · 2023-04-25 · in Deep Neural Networks from Scratch in Zig
  7. It would be cool too see a port of llama.cpp and ggml to Zig https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/ggml
    comment by remorses · 2023-04-25 · in Deep Neural Networks from Scratch in Zig
  8. …other changes and I bet you'd get something approaching JSON out. [1]: here's the line to change: https://github.com/ggerganov/llama.cpp/blob/c4fe84fb0d28851a... see the bit on line 317-319 about how it ignores the end-of-sequence token by…
    comment by gcr · 2023-04-24 · in Ask HN: People who were laid off or quit recently, how are you doing?
  9. …preliminary support in llama.cpp, which runs llama-7B in 5GB of RAM off of CPU faster than ChatGPT. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  10. …it's the same architecture as GPT-NeoX. llama.cpp just added preliminary support three hours ago. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  11. llama.cpp has preliminary support already. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  12. Issue to add support for these models to llama.cpp: https://github.com/ggerganov/llama.cpp/issues/1063
    comment by dmm · 2023-04-19 · in StableLM: A new open-source language model
  13. …A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34 There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for…
    comment by lhl · 2023-04-17 · in MiniGPT-4
  14. …if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows…
    comment by lhl · 2023-04-17 · in MiniGPT-4
  15. …* 0.5 byte = 15GB of VRAM (plus a GB or few of other overhead) For more real world discussion see https://github.com/ggerganov/llama.cpp/issues/13
    comment by mk_stjames · 2023-04-17 · in MiniGPT-4
  16. ▲ 3
    Llama.cpp: Add GPU support to ggml
    story by mromanuk · 2023-04-12 · 0 comments
  17. It’s in the near past. https://github.com/ggerganov/llama.cpp
    comment by travisjungroth · 2023-04-11 · in The Coming of Local LLMs
  18. ▲ 37
    Show HN: TurboPilot: Copilot clone runs code completion LLM on your CPU
    story by DrRavenstein · 2023-04-10 · 4 comments
  19. …[1] https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ggerganov/llama.cpp/discussions/406 [3] https://paperswithcode.com/sota/language-modelling-on-wikite... [4] https://news.ycombinator.com/it
    comment by lhl · 2023-04-10 · in The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
  20. …a standard interface for different models that can easily be integrated and tried out. [1] https://github.com/ggerganov/llama.cpp
    comment by hrpnk · 2023-04-09 · in The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
  21. …beautiful but rapidly rotting fruit. llama.cpp will remain a single two line project (git clone https://github.com/ggerganov/llama.cpp , make -j) that will compile easily and run on anything. No external deps to pin to a particular commit…
    comment by superkuh · 2023-04-06 · in State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
  22. How so? how is this: npx dalai llama 7B npx dalai serve harder than: # build this repo git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make #For Windows and CMake, use the following command instead: cd <path_to_llama_folder>…
    comment by mastazi · 2023-04-05 · in Run LLaMA and Alpaca on your computer
  23. …when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 . Jart's archived comments: "my changes" "Here's how folks in the community have been reacting…
    comment by IAmNotACellist · 2023-04-05 · in Using mmap to make LLaMA load faster
  24. …(author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
    comment by antimatter15 · 2023-04-05 · in Using mmap to make LLaMA load faster
  25. I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do…
    comment by 514723553 · 2023-04-05 · in Using mmap to make LLaMA load faster
  26. …around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
    comment by NickGerleman · 2023-04-05 · in Using mmap to make LLaMA load faster
  27. …vicuna 13B but I haven't tried that yet (converted as in this link but for 13B instead of 7B https://github.com/ggerganov/llama.cpp#usage ) GPT4All Lora's also works, perhaps the most compelling results I've got yet in my local computer -…
    comment by muyuu · 2023-04-04 · in State-of-the-art open-source chatbot, Vicuna-13B, just released model weights
  28. …same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp
    comment by newswasboring · 2023-04-04 · in An LLM playground you can run on your laptop
  29. …amount of time on this problem and had posted an interesting-but-not-production hack to it ( https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... ) on March 17th, which they had also excitedly posted about on Twitter. This was 2…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  30. …wanted to give @slaren the recognition that they deserved in a positive way, you'd have linked to https://github.com/ggerganov/llama.cpp/issues/91#issuecommen... and signal-boosted that as the key insight that enabled the PR to land,…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  31. …then proceed to take all the credit." 2. "To understand why I did that, you have to go back here: https://github.com/ggerganov/llama.cpp/issues/91#issuecommen... jart initially created an implementation of mmap a couple of week back that…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  32. …around it. Most of the drama is contained in this thread, if you like reading this sort of stuff: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
    comment by enlyth · 2023-04-03 · in Problems that attract smart people
  33. …is using. You just git clone it to your android phone in termux and then run make and you're done. https://github.com/ggerganov/llama.cpp Assuming you have the model file downloaded (you can use wget to download it) these are the…
    comment by MacsHeadroom · 2023-04-03 · in Llama.cpp 30B runs with only 6GB of RAM now
  34. > basically re-wrote it so that she could take credit for it. compare the actual PR: https://github.com/ggerganov/llama.cpp/pull/613 You'll note that it includes the original commits from Slaren , explicitly mentions the collaboration with…
    comment by detaro · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  35. …if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero…
    comment by jart · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  36. …and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discu...
    comment by fastball · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  37. …day: https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613
    comment by simonw · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  38. ▲ 86
    Revert for jart’s llama.cpp MMAP miracles
    story by mmoustafa · 2023-04-02 · 86 comments
  39. ▲ 31
    30B model now needs only 5.8GB of RAM? How?
    story by olalonde · 2023-04-01 · 11 comments
  40. …when you type "python3". I did the following: 1. Create a new working directory. 2. git clone https://github.com/ggerganov/llama.cpp 3. Download the latest release from https://github.com/ggerganov/llama.cpp/releases (note the CPU…
    comment by breput · 2023-04-01 · in Llama.cpp 30B runs with only 6GB of RAM now
  41. …to be used at evaluation time [...] Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discu...
    comment by heap_perms · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  42. …OK with the extra memory. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e... [2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...
    comment by bakkoting · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  43. Some results here: https://github.com/ggerganov/llama.cpp/discussions/406 tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B.…
    comment by bakkoting · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  44. Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after…
    comment by jart · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now
  45. ▲ 1,311
    Llama.cpp 30B runs with only 6GB of RAM now
    story by msoad · 2023-03-31 · 414 comments
  46. …tps://github.com/NouamaneTazi/bloomz.cpp ) compares to the 7B parameter Alpaca model with llama.cpp ( https://github.com/ggerganov/llama.cpp )? I have the latter working on a M1 Macbook Air with very good results for what it is. Curious if…
    comment by JasonZ2 · 2023-03-31 · in Could you train a ChatGPT-beating model for $85k and run it in a browser?
  47. Hi. I wrote the weights file format that llama.cpp uses, as of yesterday https://github.com/ggerganov/llama.cpp/pull/613 What can I do to assist you getting these deltas ready?
    comment by jart · 2023-03-31 · in Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality
  48. llama.cpp is also MIT https://github.com/ggerganov/llama.cpp previously discussed here https://news.ycombinator.com/item?id=35100086 and one of the rust wrapper: https://news.ycombinator.com/item?id=35171527 (also MIT)
    comment by homarp · 2023-03-28 · in Show HN: A fully open-source (Apache 2.0)implementation of llama
  49. …an M1 Macbook Air with 8gig RAM. Gerganov shows the Whisper model running on an Android phone. [0] https://github.com/ggerganov/llama.cpp
    comment by wsgeorge · 2023-03-28 · in ChatGPT works seamlessly with photos and files, enabling numerous new use cases
  50. …trend since the closest nodes of a global delivery network could be several countries away. [0] https://github.com/ggerganov/llama.cpp/discussions/205 [1] https://medium.com/sort-of-like-a-tech-diary/consumer-ai-is-...
    comment by wsgeorge · 2023-03-28 · in Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models