Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …the model format. - Both of these combined is what llama.cpp does, via @ochafik, in inter alia, https://github.com/ggml-org/llama.cpp/pull/9639 . - ollama isn't plugged into this system AFAIK To OP's question, specifying a format in the…
    comment by refulgentis · 2025-03-27 · in Gemma3 Function Calling
  2. I tried to execute the PoC by running the following: git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp git checkout c0d4843225eed38903ea71ef302a02fa0b27f048 # Checkout a revision prior to the exploit fix in…
    comment by yamrzou · 2025-03-24 · in Heap-overflowing Llama.cpp to RCE
  3. …another handcrafted Siglip2 NN process. Interesting—they just figured out how to handle Siglip1. https://github.com/ggml-org/llama.cpp/pull/12344/commits/631...
    comment by changtimwu · 2025-03-13 · in Gemma 3 Technical Report [pdf]
  4. …single-socket benchmarks, and includes some optimizations. Just thought it was interesting. [1]: https://github.com/ggml-org/llama.cpp/discussions/11733
    comment by magicalhippo · 2025-03-10 · in Llama.cpp AI Performance with the GeForce RTX 5090 Review
  5. The implementation's here: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/g...
    comment by regularfry · 2025-03-10 · in Llama.cpp AI Performance with the GeForce RTX 5090 Review
  6. Looks like he bases the benchmarks off of https://github.com/ggml-org/llama.cpp/discussions/4167 I get why he calls it a simulator, as it can simulate token output. It's an important aspect for evaluating use case if you need to get a…
    comment by FloatArtifact · 2025-03-06 · in Apple unveils new Mac Studio
  7. …and DDR5 might make a difference in both performance and cost , Gen 4 or Gen 5 have 8 or 9 t/s https://github.com/ggml-org/llama.cpp/discussions/11733 ) 2. AMD Epyc + 4090 or 5090 running KTransformers (over 10 t/s decode ?…
    comment by wolfgangK · 2025-03-06 · in Apple M3 Ultra
  8. Googling for what you ask has found immediately this discussion: https://github.com/ggml-org/llama.cpp/discussions/11733 about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems. While it was rather tricky, after many…
    comment by adrian_b · 2025-03-05 · in Apple M3 Ultra
  9. …with bf16 and fp8, and the latter was only 2 Elo points below, well within measurement error. [0]: https://github.com/ggml-org/llama.cpp/blob/master/examples/q... [1]: https://github.com/ggml-org/llama.cpp/discussions/5063#discu... [2]:…
    comment by espadrine · 2025-03-05 · in Apple unveils new Mac Studio
  10. x2 speed increase for ggml by optimizing SIMD: https://github.com/ggml-org/llama.cpp/pull/11453 "99% written by DeepSeek-R1" according to the author.
    comment by CamperBob2 · 2025-02-24 · in DeepSeek Open Source FlashMLA – MLA Decoding Kernel for Hopper GPUs
  11. …Pi people don't care, lol. [1] https://github.com/deepseek-ai/DeepSeek-R1#distilled-model-e... [2] https://github.com/ggerganov/llama.cpp/pull/9639
    comment by tucnak · 2025-02-15 · in Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
  12. …You could mimic either. [1] https://github.com/zed-industries/zed/tree/be830742439f531e8... [2] https://github.com/ggerganov/llama.cpp/blob/master/examples/... [3] https://github.com/zed-industries/zed/tree/be830742439f531e8...
    comment by spmurrayzzz · 2025-02-14 · in Zed now predicts your next edit with Zeta, our new open model
  13. …https://nitter.poast.org/ggerganov/status/188461277009384272... The same on Nvidia (various models) https://github.com/ggerganov/llama.cpp/issues/11474 [1] this is a the model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
    comment by un_ess · 2025-02-11 · in Building a personal, private AI computer on a budget
  14. …But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage o
    comment by kamranjon · 2025-02-11 · in Building a personal, private AI computer on a budget
  15. …An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by sofixa · 2025-02-11 · in Building a personal, private AI computer on a budget
  16. …to get good numbers. The theoretical limit should be higher, but it's not yet. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/11474#issuecom...
    comment by deoxykev · 2025-02-03 · in How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server
  17. …a grounded idea of what models can handle. Mainly - tool calling support just merged in llama.cpp ( https://github.com/ggerganov/llama.cpp/pull/9639 ) this week, and it's been a fun exercise to put local LLMs through the wringer to see how…
    comment by horsawlarway · 2025-02-03 · in Goose: An open-source, extensible AI agent that goes beyond code suggestions
  18. …template library ( https://github.com/google/minja ) just to add tool call support to llama.cpp ( https://github.com/ggerganov/llama.cpp/pull/9639 ). (*): I mean, technically my employer's :-D*
    comment by ochafik · 2025-02-01 · in Ollama are 'try[ing to] achieve vendor lock-in'
  19. …they use code from other repositories that permit such a thing is tilting at windmills, IMHO. [0] https://github.com/ggerganov/llama.cpp/blob/master/LICENSE [1] https://en.wikipedia.org/wiki/MIT_License [2]…
    comment by davely · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  20. It's in one of the collapsed comments. Quoted below: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme... So Ollama are basically forking a little bit of everything to try and achieve vendor lock-in. Some examples: The Ollama…
    comment by lioeters · 2025-01-31 · in Ollama are 'try[ing to] achieve vendor lock-in'
  21. …same atm. Not sure about the plans for RamaLama, but it might be something worth to consider. [0] https://github.com/ggerganov/llama.cpp/issues/7252
    comment by ggerganov · 2025-01-31 · in RamaLama
  22. …work, it's just not the kind of work that hackers who hang out on forums like this appreciate. [0] https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  23. This is the point of it: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
    comment by eigenvalue · 2025-01-31 · in RamaLama
  24. …and others. Is that inaccurate, or something different? [edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8010
    comment by Gracana · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  25. …llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp
    comment by woadwarrior01 · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  26. https://github.com/ggerganov/llama.cpp/discussions/9565 Discussions under https://news.ycombinator.com/item?id=40693391 (I recommend doing a search yourself first) Basically, if you know how to use a computer, you can use Ollama (almost).…
    comment by n144q · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  27. Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
    comment by Kubuxu · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  28. The answer to your question is yes. There is an open issue with llama.cpp about this very thing: https://github.com/ggerganov/llama.cpp/issues/11333 The TLDR is that llama.cpp’s NUMA support is suboptimal, which is hurting performance…
    comment by ryao · 2025-01-29 · in Complete hardware and software setup for running Deepseek-R1 locally
  29. …Additionally, llama.cpp’s NUMA support is suboptimal, so he is likely taking a performance hit: https://github.com/ggerganov/llama.cpp/issues/11333 When llama.cpp fixes its NUMA support, using two sockets should be no worse than using one…
    comment by ryao · 2025-01-29 · in Complete hardware and software setup for running Deepseek-R1 locally
  30. …replaced by the model. The heck, even your SIMD expert now seems to be replaced by the model too ( https://github.com/ggerganov/llama.cpp/pull/11453/files ). Those companies will no longer need M + N + ... engineers to create the business…
    comment by menaerus · 2025-01-29 · in Promising results from DeepSeek R1 for code
  31. …of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271. [0] https://github.com/ggerganov/llama.cpp/pull/11453/files
    comment by mjr00 · 2025-01-28 · in Promising results from DeepSeek R1 for code
  32. > vs most things related to hardware or low level work. counter point: https://github.com/ggerganov/llama.cpp/pull/11453 > This PR provides a big jump in speed for WASM by leveraging SIMD instructions for qX_K_q8_K and qX_0_q8_0 dot…
    comment by NitpickLawyer · 2025-01-27 · in I trusted an LLM, now I'm on day 4 of an afternoon project
  33. …nice idea. Does anyone know if anyone else does something similar? [0] https://www.tabbyml.com/ [1] https://github.com/ggerganov/llama.cpp/pull/9787#issue-25729... "global context onwards" [2] https://tabby.tabbyml.com/docs/administration/c…
    comment by mijoharas · 2025-01-24 · in Llama.vim – Local LLM-assisted text completion
  34. …has improved compared to what is seen in the README videos thanks to client-side caching. [0] - https://github.com/ggerganov/llama.cpp/pull/9787 [1] - https://github.com/ggml-org/llama.vscode
    comment by ggerganov · 2025-01-23 · in Llama.vim – Local LLM-assisted text completion
  35. …image processing requirements AFIAK. [1]: https://github.com/ollama/ollama/releases/tag/v0.4.0 [2]: https://github.com/ggerganov/llama.cpp/issues/9643
    comment by magicalhippo · 2025-01-12 · in Phi 4 available on Ollama
  36. …output to tokens with the desired prefix, probably with some mechanism similar to how this works: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by do_not_redeem · 2025-01-11 · in Character Prefix Conditioning
  37. …runs reasonably well on a 24GB GPU (7900XTX or 4090) and 64GB of regular RAM. Software: https://github.com/ggerganov/llama.cpp .
    comment by skirmish · 2024-12-26 · in GPT-5 is behind schedule
  38. Compare performance on various Macs here as it gets updated: https://github.com/ggerganov/llama.cpp/discussions/4167 OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling…
    comment by Terretta · 2024-12-15 · in Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning
  39. …offers significant speed improvements, which I understand can sometimes be shortcoming of GBNF (e.g https://github.com/ggerganov/llama.cpp/issues/4218 ).
    comment by mwieler · 2024-12-08 · in Structured Outputs with Ollama
  40. …[1] https://github.com/guidance-ai/llguidance/blob/main/parser/s... [2] https://github.com/ggerganov/llama.cpp/pull/10224 [3] https://github.com/guidance-ai/llgtrt
    comment by mmoskal · 2024-12-07 · in Structured Outputs with Ollama
  41. If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by rdescartes · 2024-12-07 · in Structured Outputs with Ollama
  42. …line in the end? [1] - https://github.com/ollama/ollama/commit/1bdab9fdb19f8a8c73ed... [2] - since https://github.com/ggerganov/llama.cpp/commit/bcc0eb4591bec5...
    comment by wokwokwok · 2024-12-05 · in Bringing K/V context quantisation to Ollama
  43. …a good way to get confidence scores in llama.cpp (Been tracking this git request to get log_probs https://github.com/ggerganov/llama.cpp/issues/6423 )
    comment by inLewOf · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  44. …l-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true # build llama.cpp git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang++-17 cmake -S. -Bbuild \ -DGGML_HIPBLAS=ON \…
    comment by slavik81 · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  45. You can get a release binary from https://github.com/ggerganov/llama.cpp/releases too.
    comment by xyc · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  46. …from the fp16 variant. [1] https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF [2] https://github.com/ggerganov/llama.cpp
    comment by johndough · 2024-11-10 · in OpenCoder: Open Cookbook for Top-Tier Code Large Language Models
  47. No need to use Ollama. LLama.cpp has its own OpenAI-compatible server[0] and it works great. [0] https://github.com/ggerganov/llama.cpp#web-server
    comment by milansuk · 2024-10-30 · in Go library for in-process vector search and embeddings with llama.cpp
  48. …similar experience - but did find I got better output by forcing them to adhere to a fixed grammar: https://github.com/ggerganov/llama.cpp/tree/master/grammars For context, I was playing with a script to bulk download podcasts, transcribe…
    comment by wswope · 2024-10-24 · in Quantized Llama models with increased speed and a reduced memory footprint
  49. …There are a variety of other LLM inference implementations that can run on CPU as well. [0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su... [1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...
    comment by kkielhofner · 2024-10-18 · in Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?
  50. …non-deterministic (maybe stochastic) output, but maybe we will see it in the next few years. [1] https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Lockal · 2024-10-18 · in The Prompt() Function: Use the Power of LLMs with SQL