Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …as well now that you could point your UI at (if you wanted to run leaner w/out ollama). https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by spmurrayzzz · 2024-02-08 · in OpenAI compatibility
  2. I recently found out about llama-cpp's official "Server" function: https://github.com/ggerganov/llama.cpp/blob/master/examples/... Works like a charm for my (simple) use case. I'm running it on an Always Free Oracle Ampere A1 instance with…
    comment by smoldesu · 2024-02-07 · in Ask HN: If you're running an LLM locally, what are you using?
  3. …is very much focused on CUDA; llama.cpp has an open issue about Metal-accelerated training: https://github.com/ggerganov/llama.cpp/issues/3799 - but no work on it so far. This is likely because training at any significant sizes requires…
    comment by int_19h · 2024-01-29 · in Meta AI releases Code Llama 70B
  4. There's https://github.com/saharNooby/rwkv.cpp , which related-ish[0] to ggml/llama.cpp [0]: https://github.com/ggerganov/llama.cpp/issues/846
    comment by kristjansson · 2024-01-29 · in Eagle 7B: Soaring past Transformers
  5. Llama.cpp has an interactive mode, but I don't think text-generation-webui uses it. https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by sp332 · 2024-01-27 · in Brave Leo now uses Mixtral 8x7B as default
  6. Nice graphs here: https://github.com/ggerganov/llama.cpp/pull/1684 So for example, 2 bit version of the 30B is much worse than the original, but still better than the 13B model. Also, there are lots of extra details, eg, not all of the…
    comment by DougBTX · 2024-01-27 · in Brave Leo now uses Mixtral 8x7B as default
  7. +1 to this. I’ve been working with their research teams on this and credit due indeed. Some context: https://github.com/ggerganov/llama.cpp/issues/2555#issuecomm... https://github.com/ggerganov/llama.cpp/discussions/3965
    comment by kiratp · 2024-01-20 · in Efficient LLM inference solution on Intel GPU
  8. …by memory size and bandwidth (hence the aggressive quantization), rather than raw compute. [1] https://github.com/ggerganov/llama.cpp/discussions/844
    comment by Retr0id · 2024-01-18 · in ARM's "Blackhawk" CPU Is a Plan to Have the Best Smartphone CPU Core
  9. lamma.cpp allows you to restrict the output such that it would always generate valid JSON https://github.com/ggerganov/llama.cpp#constrained-output-wi...
    comment by abadugu · 2024-01-17 · in Building a fully local LLM voice assistant to control my smart home
  10. llama.cpp supports custom grammars to constrain inference. maybe this is a helpful starting point? https://github.com/ggerganov/llama.cpp/tree/master/grammars
    comment by khimaros · 2024-01-14 · in Building a fully local LLM voice assistant to control my smart home
  11. Running AI models on edge devices is already here. This is just one example https://github.com/ggerganov/llama.cpp With Swift language bindings it would be quite smooth to work towards.
    comment by whirlwin · 2024-01-13 · in Apple hiring compiler developers for improving Swift / C++ interoperability
  12. …for being < 3B parameter models. They can run on a phone, and are pretty snappy. Benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4508 I don't see them taking over general purpose chat/query use cases, but fined tuned to a…
    comment by scosman · 2024-01-12 · in Phi-2: Self-Extend Boosts Performance, Extends Context to 8k Without Training
  13. …surprisingly well for Mixtral 8x7B Instruct with 2.10 bits per weight (12.3 GB model size). [1] https://github.com/ggerganov/llama.cpp/pull/4773
    comment by bekantan · 2024-01-09 · in Mixtral 8x7B: A sparse Mixture of Experts language model
  14. …per.apple.com/documentation/metal/gpu_devices_... [8] https://github.com/ml-explore/mlx/issues/18 [9] https://github.com/ggerganov/llama.cpp/issues/1714#issuecomm... [10] https://testflight.apple.com/join/ERFxInZg
    comment by brittlewis12 · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  15. …on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface. https://github.com/ggerganov/llama.cpp UI projects in description. https://huggingface.co/TheBloke A closed source option is LMStudio.…
    comment by kiratp · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  16. almost certainly - edit: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by castles · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  17. Here is how to do that on Android: https://github.com/ggerganov/llama.cpp/#android
    comment by alekseiprokopev · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  18. …[2] https://www.amazon.com/Zalman-Computer-Workstation-Preinstal... [3] https://github.com/ggerganov/llama.cpp/discussions/4167 [4] https://kubito.dev/posts/llama-nvidia-3070-ti-benchmarks/#pe...
    comment by cwaffles · 2024-01-06 · in M3 CPU cores have become more versatile
  19. …reasons the project can move so fast and be such an essential project in the LLM scene. [1]: https://github.com/ggerganov/llama.cpp/graphs/contributors
    comment by WiSaGaN · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  20. If anyone is interested in avoiding bloat, llama.cpp already includes an OpenAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Art9681 · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  21. …other inference backends directly themselves. llama.cpp already includes an openAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Art9681 · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  22. …to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216 , https://github.com/ggerganov/llama.cpp/blob/master/examples/... ). ...but a production…
    comment by wokwokwok · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  23. Performance of llama.cpp on Apple Silicon M-series: https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by falava · 2024-01-05 · in Best Mac for Large Language Models
  24. https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b…
    comment by eightysixfour · 2024-01-05 · in A simulation of me: fine-tuning an LLM on 240k text messages
  25. Hi, I haven't tried this myself, but it seems there's a way? https://github.com/ggerganov/llama.cpp/blob/master/examples/... The call takes a "stream" boolean: stream: It allows receiving each predicted token in real-time instead of…
    comment by exe34 · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs
  26. …support to llama.cpp back in March, specifically so I could build things like Emacs Copilot. See: https://github.com/ggerganov/llama.cpp/pull/613 Recently I've been working with Mozilla to create llamafile, so that using llama.cpp can be…
    comment by jart · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs
  27. Instructions: 1. Install llama.cpp from https://github.com/ggerganov/llama.cpp . Alternatively, install https://github.com/oobabooga/text-generation-webui 2. Go to https://huggingface.co/TheBloke and search for GGUF. Download and put the…
    comment by patrakov · 2023-12-29 · in AirLLM enables 8GB MacBook run 70B LLM
  28. …makefile, which is an example of an excellent cross platform makefile that works very well: https://github.com/ggerganov/llama.cpp/blob/master/Makefile ^ this is what many real Makefiles look like, and this one is excellent and well…
    comment by wokwokwok · 2023-12-27 · in Using Make – writing less Makefile
  29. …oke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048…
    comment by rightbyte · 2023-12-25 · in 2023: The Year of AI
  30. …by llamafile when I can just install llama.cpp? (the web interface is also part of llama.cpp https://github.com/ggerganov/llama.cpp/pull/1998 ) Moreover, people with GPUs will likely want to use something with better performance than…
    comment by jokethrowaway · 2023-12-21 · in Llamafile: Bringing LLMs to the people, and to your own computer
  31. …valid JSON (that matches your schema!) and no extraneous text. Something potentially helpful here: https://github.com/ggerganov/llama.cpp/discussions/2494 If you fine-tuned a base model (like the one in the article) on various inputs and…
    comment by coder543 · 2023-12-21 · in Mistral 7B Fine-Tune Optimized
  32. …the llama.cpp main.cpp and other normal llama.cpp code. It's a fork. It is directly comparable. https://github.com/ggerganov/llama.cpp/pull/4543 [Review] Merge PowerInfer with llama.cpp mainline #4543 https://github.com/ggerganov/llama.cpp/…
    comment by superkuh · 2023-12-20 · in High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUs
  33. …Unless you use this hack [2]. [1] https://developer.apple.com/forums/thread/732035 [2] https://github.com/ggerganov/llama.cpp/discussions/2182#disc...
    comment by indexerror · 2023-12-20 · in LLM in a Flash: Efficient LLM Inference with Limited Memory
  34. This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543 One thing that caught my interest is this line from their readme: > PowerInfer exploits such…
    comment by chsasank · 2023-12-20 · in PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]
  35. Sure, this is common e.g. https://github.com/ggerganov/llama.cpp/pull/613
    comment by fancy_pantser · 2023-12-20 · in LLM in a Flash: Efficient LLM Inference with Limited Memory
  36. Try https://github.com/ggerganov/llama.cpp Builds very quickly with make. But if it's slow when you try it then make sure to enable any flags related to CUDA and then try the build again. A key parameter is the one that tells it how many…
    comment by ilaksh · 2023-12-19 · in Perplexity Labs Playground
  37. Sampling methods also affect this. Have you tried min_p sampling? https://github.com/ggerganov/llama.cpp/pull/3841#issuecommen...
    comment by vczf · 2023-12-19 · in An In-depth Look at Gemini's Language Abilities
  38. …[0]. My ~10 year old MacBook Air has 4GB of RAM, so I'm primarily interested in smaller LLMs. [0] https://github.com/ggerganov/llama.cpp
    comment by physicsgraph · 2023-12-18 · in Small offline large language model – TinyChatEngine from MIT
  39. You can look at https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... for examples of the different layers in a number of different models, and further down in the code for their implementations. tldr, yes they are very similar.…
    comment by andy99 · 2023-12-17 · in Transformers on Chips
  40. …universal macOS/iOS app[3] that does support SWA with Mistral models (using mlc-llm). [1]: https://github.com/ggerganov/llama.cpp/issues/3377 [2]: https://arxiv.org/abs/2310.06825 [3]: https://apps.apple.com/us/app/private-llm/id6448106860
    comment by woadwarrior01 · 2023-12-17 · in Run Mistral 7B on M1 Mac
  41. …easy to dive into with consumer hardware, but don't know any great papers myself Run locally: https://github.com/ggerganov/llama.cpp Quantized models: https://huggingface.co/TheBloke Explainability is under research, though I haven't seen…
    comment by meltyness · 2023-12-15 · in Ask HN: AI/ML papers to catch up with current state of AI?
  42. There is an example of training https://github.com/ggerganov/llama.cpp/tree/1f0bccb27929e261... But that's absolutely false about the Nvidia moat being only training. Llama.cpp makes it far more practical run inference on a variety of…
    comment by ilaksh · 2023-12-15 · in Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'
  43. No support in llama.cpp yet :( https://github.com/ggerganov/llama.cpp/issues/3146
    comment by gigel82 · 2023-12-13 · in Phi-2
  44. …shell script before that uses a local llm as a composable tool? I know there's plenty of stuff like https://github.com/ggerganov/llama.cpp/blob/master/examples/... where the shell script is being used to supply all the llama.cpp arguments…
    comment by jart · 2023-12-13 · in Bash one-liners for LLMs
  45. Note that running these GGUF models currently requires a forked version of llama.cpp: https://github.com/ggerganov/llama.cpp/pull/4406 The GGUF handling for Mistral's mixture of experts hasn't been finalized yet. TheBloke and ggerganov and…
    comment by loudmax · 2023-12-11 · in Mixtral of experts
  46. +1 on CogVLM. https://github.com/THUDM/CogVLM Recent discussions: https://github.com/ggerganov/llama.cpp/discussions/4350 It is the best open source vision language model out there that I'm aware of that's most comparable to gpt4v. Beats…
    comment by stevenhuang · 2023-12-11 · in The industries AI is disrupting are not lucrative
  47. …that it can support is expanding [2]. Folks have also written services [3] that wrap around it. [1] https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python
    comment by curious_cat_163 · 2023-12-10 · in Mistral-8x7B-Chat
  48. what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
    comment by rgbrgb · 2023-12-10 · in Mistral-8x7B-Chat
  49. There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381
    comment by kristianp · 2023-12-10 · in Mistral-8x7B-Chat
  50. …implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h
    comment by westurner · 2023-12-10 · in Android adds ggml lib to AICore