Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. Nice graphs here: https://github.com/ggerganov/llama.cpp/pull/1684 So for example, 2 bit version of the 30B is much worse than the original, but still better than the 13B model. Also, there are lots of extra details, eg, not all of the…
    comment by DougBTX · 2024-01-27 · in Brave Leo now uses Mixtral 8x7B as default
  2. ▲ 2
    Llama.cpp incoming backends: Vulkan, Kompute, SYCL
    story by irusensei · 2024-01-27 · 0 comments
  3. +1 to this. I’ve been working with their research teams on this and credit due indeed. Some context: https://github.com/ggerganov/llama.cpp/issues/2555#issuecomm... https://github.com/ggerganov/llama.cpp/discussions/3965
    comment by kiratp · 2024-01-20 · in Efficient LLM inference solution on Intel GPU
  4. …by memory size and bandwidth (hence the aggressive quantization), rather than raw compute. [1] https://github.com/ggerganov/llama.cpp/discussions/844
    comment by Retr0id · 2024-01-18 · in ARM's "Blackhawk" CPU Is a Plan to Have the Best Smartphone CPU Core
  5. lamma.cpp allows you to restrict the output such that it would always generate valid JSON https://github.com/ggerganov/llama.cpp#constrained-output-wi...
    comment by abadugu · 2024-01-17 · in Building a fully local LLM voice assistant to control my smart home
  6. llama.cpp supports custom grammars to constrain inference. maybe this is a helpful starting point? https://github.com/ggerganov/llama.cpp/tree/master/grammars
    comment by khimaros · 2024-01-14 · in Building a fully local LLM voice assistant to control my smart home
  7. Running AI models on edge devices is already here. This is just one example https://github.com/ggerganov/llama.cpp With Swift language bindings it would be quite smooth to work towards.
    comment by whirlwin · 2024-01-13 · in Apple hiring compiler developers for improving Swift / C++ interoperability
  8. …for being < 3B parameter models. They can run on a phone, and are pretty snappy. Benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4508 I don't see them taking over general purpose chat/query use cases, but fined tuned to a…
    comment by scosman · 2024-01-12 · in Phi-2: Self-Extend Boosts Performance, Extends Context to 8k Without Training
  9. ▲ 2
    Llama.cpp: Self-Extend Support
    story by tosh · 2024-01-09 · 0 comments
  10. …surprisingly well for Mixtral 8x7B Instruct with 2.10 bits per weight (12.3 GB model size). [1] https://github.com/ggerganov/llama.cpp/pull/4773
    comment by bekantan · 2024-01-09 · in Mixtral 8x7B: A sparse Mixture of Experts language model
  11. …per.apple.com/documentation/metal/gpu_devices_... [8] https://github.com/ml-explore/mlx/issues/18 [9] https://github.com/ggerganov/llama.cpp/issues/1714#issuecomm... [10] https://testflight.apple.com/join/ERFxInZg
    comment by brittlewis12 · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  12. …on CPU. You can use any of the “Q4_M” “GGUF” models that TheBloke puts out on Huggingface. https://github.com/ggerganov/llama.cpp UI projects in description. https://huggingface.co/TheBloke A closed source option is LMStudio.…
    comment by kiratp · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  13. almost certainly - edit: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by castles · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  14. Here is how to do that on Android: https://github.com/ggerganov/llama.cpp/#android
    comment by alekseiprokopev · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros
  15. ▲ 5
    Llama.cpp: SOTA 2-bit quants
    story by tosh · 2024-01-07 · 0 comments
  16. …[2] https://www.amazon.com/Zalman-Computer-Workstation-Preinstal... [3] https://github.com/ggerganov/llama.cpp/discussions/4167 [4] https://kubito.dev/posts/llama-nvidia-3070-ti-benchmarks/#pe...
    comment by cwaffles · 2024-01-06 · in M3 CPU cores have become more versatile
  17. …reasons the project can move so fast and be such an essential project in the LLM scene. [1]: https://github.com/ggerganov/llama.cpp/graphs/contributors
    comment by WiSaGaN · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  18. If anyone is interested in avoiding bloat, llama.cpp already includes an OpenAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Art9681 · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  19. …other inference backends directly themselves. llama.cpp already includes an openAI compatible API: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Art9681 · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  20. …to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216 , https://github.com/ggerganov/llama.cpp/blob/master/examples/... ). ...but a production…
    comment by wokwokwok · 2024-01-06 · in Nitro: A fast, lightweight inference server with OpenAI-Compatible API
  21. Performance of llama.cpp on Apple Silicon M-series: https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by falava · 2024-01-05 · in Best Mac for Large Language Models
  22. https://github.com/ggerganov/llama.cpp/pull/1684 You are broadly better off running a larger, more quantized model, than a smaller model at a higher precision. A 65b parameter model with 2bit quantization still outperforms an fp16 30b…
    comment by eightysixfour · 2024-01-05 · in A simulation of me: fine-tuning an LLM on 240k text messages
  23. Hi, I haven't tried this myself, but it seems there's a way? https://github.com/ggerganov/llama.cpp/blob/master/examples/... The call takes a "stream" boolean: stream: It allows receiving each predicted token in real-time instead of…
    comment by exe34 · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs
  24. …support to llama.cpp back in March, specifically so I could build things like Emacs Copilot. See: https://github.com/ggerganov/llama.cpp/pull/613 Recently I've been working with Mozilla to create llamafile, so that using llama.cpp can be…
    comment by jart · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs
  25. ▲ 2
    K-Quants
    story by tosh · 2023-12-29 · 0 comments
  26. Instructions: 1. Install llama.cpp from https://github.com/ggerganov/llama.cpp . Alternatively, install https://github.com/oobabooga/text-generation-webui 2. Go to https://huggingface.co/TheBloke and search for GGUF. Download and put the…
    comment by patrakov · 2023-12-29 · in AirLLM enables 8GB MacBook run 70B LLM
  27. …makefile, which is an example of an excellent cross platform makefile that works very well: https://github.com/ggerganov/llama.cpp/blob/master/Makefile ^ this is what many real Makefiles look like, and this one is excellent and well…
    comment by wokwokwok · 2023-12-27 · in Using Make – writing less Makefile
  28. …oke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048…
    comment by rightbyte · 2023-12-25 · in 2023: The Year of AI
  29. …by llamafile when I can just install llama.cpp? (the web interface is also part of llama.cpp https://github.com/ggerganov/llama.cpp/pull/1998 ) Moreover, people with GPUs will likely want to use something with better performance than…
    comment by jokethrowaway · 2023-12-21 · in Llamafile: Bringing LLMs to the people, and to your own computer
  30. ▲ 3
    CUDA: Faster Mixtral Prompt Processing
    story by tosh · 2023-12-21 · 0 comments
  31. …valid JSON (that matches your schema!) and no extraneous text. Something potentially helpful here: https://github.com/ggerganov/llama.cpp/discussions/2494 If you fine-tuned a base model (like the one in the article) on various inputs and…
    comment by coder543 · 2023-12-21 · in Mistral 7B Fine-Tune Optimized
  32. …the llama.cpp main.cpp and other normal llama.cpp code. It's a fork. It is directly comparable. https://github.com/ggerganov/llama.cpp/pull/4543 [Review] Merge PowerInfer with llama.cpp mainline #4543 https://github.com/ggerganov/llama.cpp/…
    comment by superkuh · 2023-12-20 · in High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUs
  33. …Unless you use this hack [2]. [1] https://developer.apple.com/forums/thread/732035 [2] https://github.com/ggerganov/llama.cpp/discussions/2182#disc...
    comment by indexerror · 2023-12-20 · in LLM in a Flash: Efficient LLM Inference with Limited Memory
  34. This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: https://github.com/ggerganov/llama.cpp/pull/4543 One thing that caught my interest is this line from their readme: > PowerInfer exploits such…
    comment by chsasank · 2023-12-20 · in PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]
  35. Sure, this is common e.g. https://github.com/ggerganov/llama.cpp/pull/613
    comment by fancy_pantser · 2023-12-20 · in LLM in a Flash: Efficient LLM Inference with Limited Memory
  36. ▲ 100
    Performance of llama.cpp on Apple Silicon A-series
    story by mobilio · 2023-12-19 · 41 comments
  37. ▲ 3
    Llama.cpp: Support for Phi-2
    story by tosh · 2023-12-19 · 0 comments
  38. Try https://github.com/ggerganov/llama.cpp Builds very quickly with make. But if it's slow when you try it then make sure to enable any flags related to CUDA and then try the build again. A key parameter is the one that tells it how many…
    comment by ilaksh · 2023-12-19 · in Perplexity Labs Playground
  39. Sampling methods also affect this. Have you tried min_p sampling? https://github.com/ggerganov/llama.cpp/pull/3841#issuecommen...
    comment by vczf · 2023-12-19 · in An In-depth Look at Gemini's Language Abilities
  40. …[0]. My ~10 year old MacBook Air has 4GB of RAM, so I'm primarily interested in smaller LLMs. [0] https://github.com/ggerganov/llama.cpp
    comment by physicsgraph · 2023-12-18 · in Small offline large language model – TinyChatEngine from MIT
  41. You can look at https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... for examples of the different layers in a number of different models, and further down in the code for their implementations. tldr, yes they are very similar.…
    comment by andy99 · 2023-12-17 · in Transformers on Chips
  42. …universal macOS/iOS app[3] that does support SWA with Mistral models (using mlc-llm). [1]: https://github.com/ggerganov/llama.cpp/issues/3377 [2]: https://arxiv.org/abs/2310.06825 [3]: https://apps.apple.com/us/app/private-llm/id6448106860
    comment by woadwarrior01 · 2023-12-17 · in Run Mistral 7B on M1 Mac
  43. …easy to dive into with consumer hardware, but don't know any great papers myself Run locally: https://github.com/ggerganov/llama.cpp Quantized models: https://huggingface.co/TheBloke Explainability is under research, though I haven't seen…
    comment by meltyness · 2023-12-15 · in Ask HN: AI/ML papers to catch up with current state of AI?
  44. There is an example of training https://github.com/ggerganov/llama.cpp/tree/1f0bccb27929e261... But that's absolutely false about the Nvidia moat being only training. Llama.cpp makes it far more practical run inference on a variety of…
    comment by ilaksh · 2023-12-15 · in Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'
  45. ▲ 3
    QMoE Support for Mixtral
    story by tosh · 2023-12-14 · 0 comments
  46. No support in llama.cpp yet :( https://github.com/ggerganov/llama.cpp/issues/3146
    comment by gigel82 · 2023-12-13 · in Phi-2
  47. …shell script before that uses a local llm as a composable tool? I know there's plenty of stuff like https://github.com/ggerganov/llama.cpp/blob/master/examples/... where the shell script is being used to supply all the llama.cpp arguments…
    comment by jart · 2023-12-13 · in Bash one-liners for LLMs
  48. ▲ 2
    Llama: Add Mixtral Support
    story by tosh · 2023-12-11 · 0 comments
  49. Note that running these GGUF models currently requires a forked version of llama.cpp: https://github.com/ggerganov/llama.cpp/pull/4406 The GGUF handling for Mistral's mixture of experts hasn't been finalized yet. TheBloke and ggerganov and…
    comment by loudmax · 2023-12-11 · in Mixtral of experts
  50. +1 on CogVLM. https://github.com/THUDM/CogVLM Recent discussions: https://github.com/ggerganov/llama.cpp/discussions/4350 It is the best open source vision language model out there that I'm aware of that's most comparable to gpt4v. Beats…
    comment by stevenhuang · 2023-12-11 · in The industries AI is disrupting are not lucrative