Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerganov/llama.cpp
    comment by samwillis · 2023-05-31 · in Ask HN: What's the best self hosted/local alternative to GPT-4?
  2. …useful to have GPU support. As a baseline, this can support anything that llama.cpp. Relevant link: https://github.com/ggerganov/llama.cpp#blas-build
    comment by ybu · 2023-05-24 · in Show HN: a desktop app to discover, download, and run offline LLMs
  3. …on https://www.reddit.com/r/LocalLLaMA . The best way for you to run the model is probably through https://github.com/ggerganov/llama.cpp . This is a plain C/C++ implementation that can run LLMs pretty efficiently. It can run the LLaMA 13B…
    comment by alpark3 · 2023-05-20 · in Ask HN: Best UNCENSORED language model comparable to ChatGPT?
  4. …implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h Are the recommendable ways to cast e.g. arrow Tensors to pytorch/tensorflow?…
    comment by westurner · 2023-05-20 · in PyTorch for WebGPU
  5. Go see yourself :) https://github.com/ggerganov/llama.cpp/blob/master/examples/... There's too many schemes right now with 4_0 and 5_1 really popular between LLM geeks.
    comment by Ambix · 2023-05-18 · in Numbers every LLM developer should know
  6. https://github.com/ggerganov/llama.cpp https://huggingface.co/mosaicml/mpt-7b
    comment by T-A · 2023-05-18 · in Show HN: Smallville – Create generative agents for simulations and games
  7. …shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
    comment by waleedk · 2023-05-17 · in Numbers every LLM developer should know
  8. Taks a look at: https://github.com/ggerganov/llama.cpp
    comment by iamflimflam1 · 2023-05-17 · in A guidance language for controlling LLMs
  9. First link: https://github.com/ggerganov/llama.cpp Which in turn has the following as the first link: https://arxiv.org/abs/2302.13971 Is it really quicker to ask here than just browse content for a bit, skimming some text or even using…
    comment by capableweb · 2023-05-14 · in Run Llama 13B with a 6GB graphics card
  10. …model architecture is slightly different, based on Pythia I guess what is needed is a pythia.cpp https://github.com/ggerganov/llama.cpp/issues/742
    comment by anentropic · 2023-05-06 · in Releasing 3B and 7B RedPajama
  11. You can do it on CPU now. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  12. Benchmarks for what you can do on CPU alone. https://github.com/ggerganov/llama.cpp/issues/34 An M1 Max does 100ms per token. A 64 core threadripper about 33ms per token.
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  13. And here are the benchmarks https://github.com/ggerganov/llama.cpp/issues/34
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  14. To use with llama.cpp on CPU and 8GB RAM git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build python3 -m pip install -r requirements.txt cd models && git clone…
    comment by diimdeep · 2023-05-03 · in OpenLLaMA: An Open Reproduction of LLaMA
  15. I think llama.cpp might be easier to set up and get running. https://github.com/ggerganov/llama.cpp
    comment by jbandela1 · 2023-05-03 · in OpenLLaMA: An Open Reproduction of LLaMA
  16. There's some info here: https://github.com/ggerganov/llama.cpp/issues/1240
    comment by ac29 · 2023-04-30 · in Whisper.cpp v1.4.0
  17. …build-essential first. Congratulations, now you too, a non-developer, are running it locally. https://github.com/ggerganov/llama.cpp
    comment by PostOnce · 2023-04-30 · in A brief history of LLaMA models
  18. …of the models out in the open really have GPT-3-ish capabilities, not even llama-65b. That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable…
    comment by int_19h · 2023-04-28 · in What's up in the Python community? – April 2023
  19. …because these models are constrained by memory bandwidth and not compute (low arithmetic density) https://github.com/ggerganov/llama.cpp
    comment by remorses · 2023-04-25 · in Deep Neural Networks from Scratch in Zig
  20. It would be cool too see a port of llama.cpp and ggml to Zig https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/ggml
    comment by remorses · 2023-04-25 · in Deep Neural Networks from Scratch in Zig
  21. …other changes and I bet you'd get something approaching JSON out. [1]: here's the line to change: https://github.com/ggerganov/llama.cpp/blob/c4fe84fb0d28851a... see the bit on line 317-319 about how it ignores the end-of-sequence token by…
    comment by gcr · 2023-04-24 · in Ask HN: People who were laid off or quit recently, how are you doing?
  22. …preliminary support in llama.cpp, which runs llama-7B in 5GB of RAM off of CPU faster than ChatGPT. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  23. …it's the same architecture as GPT-NeoX. llama.cpp just added preliminary support three hours ago. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  24. llama.cpp has preliminary support already. https://github.com/ggerganov/llama.cpp/issues/1063#issuecomm...
    comment by MacsHeadroom · 2023-04-20 · in StableLM: A new open-source language model
  25. Issue to add support for these models to llama.cpp: https://github.com/ggerganov/llama.cpp/issues/1063
    comment by dmm · 2023-04-19 · in StableLM: A new open-source language model
  26. …A token is about 0.75/words, so you can see lots of people getting 4-8 words/s on their CPUs: https://github.com/ggerganov/llama.cpp/issues/34 There a lot of optimizations that can be done. Here's one w/ potentially a 15X AVX speedup for…
    comment by lhl · 2023-04-17 · in MiniGPT-4
  27. …if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows…
    comment by lhl · 2023-04-17 · in MiniGPT-4
  28. …* 0.5 byte = 15GB of VRAM (plus a GB or few of other overhead) For more real world discussion see https://github.com/ggerganov/llama.cpp/issues/13
    comment by mk_stjames · 2023-04-17 · in MiniGPT-4
  29. It’s in the near past. https://github.com/ggerganov/llama.cpp
    comment by travisjungroth · 2023-04-11 · in The Coming of Local LLMs
  30. …[1] https://github.com/facebookresearch/llama/blob/main/MODEL_CA... [2] https://github.com/ggerganov/llama.cpp/discussions/406 [3] https://paperswithcode.com/sota/language-modelling-on-wikite... [4] https://news.ycombinator.com/it
    comment by lhl · 2023-04-10 · in The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
  31. …a standard interface for different models that can easily be integrated and tried out. [1] https://github.com/ggerganov/llama.cpp
    comment by hrpnk · 2023-04-09 · in The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
  32. …beautiful but rapidly rotting fruit. llama.cpp will remain a single two line project (git clone https://github.com/ggerganov/llama.cpp , make -j) that will compile easily and run on anything. No external deps to pin to a particular commit…
    comment by superkuh · 2023-04-06 · in State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration
  33. How so? how is this: npx dalai llama 7B npx dalai serve harder than: # build this repo git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make #For Windows and CMake, use the following command instead: cd <path_to_llama_folder>…
    comment by mastazi · 2023-04-05 · in Run LLaMA and Alpaca on your computer
  34. …when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 . Jart's archived comments: "my changes" "Here's how folks in the community have been reacting…
    comment by IAmNotACellist · 2023-04-05 · in Using mmap to make LLaMA load faster
  35. …(author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
    comment by antimatter15 · 2023-04-05 · in Using mmap to make LLaMA load faster
  36. I don't get the "plagiarism/miscrediting" accusations. This was in the original PR ( https://github.com/ggerganov/llama.cpp/pull/613 ): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do…
    comment by 514723553 · 2023-04-05 · in Using mmap to make LLaMA load faster
  37. …around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711
    comment by NickGerleman · 2023-04-05 · in Using mmap to make LLaMA load faster
  38. …vicuna 13B but I haven't tried that yet (converted as in this link but for 13B instead of 7B https://github.com/ggerganov/llama.cpp#usage ) GPT4All Lora's also works, perhaps the most compelling results I've got yet in my local computer -…
    comment by muyuu · 2023-04-04 · in State-of-the-art open-source chatbot, Vicuna-13B, just released model weights
  39. …same performance as GPT-4. Look up llama.cpp[1] and all the various community generated models. [1] https://github.com/ggerganov/llama.cpp
    comment by newswasboring · 2023-04-04 · in An LLM playground you can run on your laptop
  40. …amount of time on this problem and had posted an interesting-but-not-production hack to it ( https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... ) on March 17th, which they had also excitedly posted about on Twitter. This was 2…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  41. …wanted to give @slaren the recognition that they deserved in a positive way, you'd have linked to https://github.com/ggerganov/llama.cpp/issues/91#issuecommen... and signal-boosted that as the key insight that enabled the PR to land,…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  42. …then proceed to take all the credit." 2. "To understand why I did that, you have to go back here: https://github.com/ggerganov/llama.cpp/issues/91#issuecommen... jart initially created an implementation of mmap a couple of week back that…
    comment by lhnz · 2023-04-03 · in Revert for jart’s llama.cpp MMAP miracles
  43. …around it. Most of the drama is contained in this thread, if you like reading this sort of stuff: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
    comment by enlyth · 2023-04-03 · in Problems that attract smart people
  44. …is using. You just git clone it to your android phone in termux and then run make and you're done. https://github.com/ggerganov/llama.cpp Assuming you have the model file downloaded (you can use wget to download it) these are the…
    comment by MacsHeadroom · 2023-04-03 · in Llama.cpp 30B runs with only 6GB of RAM now
  45. > basically re-wrote it so that she could take credit for it. compare the actual PR: https://github.com/ggerganov/llama.cpp/pull/613 You'll note that it includes the original commits from Slaren , explicitly mentions the collaboration with…
    comment by detaro · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  46. …if it's 4chan but it appears to be some chan. When I was working on solving the mmap() problem in https://github.com/ggerganov/llama.cpp/issues/91 (that issue tracks the full history) I originally wrote a malloc() hack to prove zero…
    comment by jart · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  47. …and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discu...
    comment by fastball · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  48. …day: https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613
    comment by simonw · 2023-04-02 · in Revert for jart’s llama.cpp MMAP miracles
  49. …when you type "python3". I did the following: 1. Create a new working directory. 2. git clone https://github.com/ggerganov/llama.cpp 3. Download the latest release from https://github.com/ggerganov/llama.cpp/releases (note the CPU…
    comment by breput · 2023-04-01 · in Llama.cpp 30B runs with only 6GB of RAM now
  50. …to be used at evaluation time [...] Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discu...
    comment by heap_perms · 2023-03-31 · in Llama.cpp 30B runs with only 6GB of RAM now