Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …llama.cpp's LICENSE: "Copyright (c) 2023-2024 The ggml authors". ollama doesn't include it. [0]: https://github.com/ggml-org/llama.cpp/blob/master/LICENSE#L3...
    comment by camgunz · 2025-05-16 · in Ollama violating llama.cpp license for over a year
  2. …license is in the ollama project as is in the llama.cpp project, is this not sufficient? llamma.cpp https://github.com/ggml-org/llama.cpp/blob/master/LICENSE ollama.cpp https://github.com/ollama/ollama/blob/main/LICENSE
    comment by gmm1990 · 2025-05-16 · in Ollama violating llama.cpp license for over a year
  3. llama.cpp clearly does not support iSWA: https://github.com/ggml-org/llama.cpp/issues/12637 Ollama does, please try it.
    comment by nolist_policy · 2025-05-16 · in Ollama's new engine for multimodal models
  4. …is written in golang so of course they can not meaningfully contribute that back to llama.cpp. [0] https://github.com/ggml-org/llama.cpp/issues/12637
    comment by nolist_policy · 2025-05-16 · in Ollama's new engine for multimodal models
  5. …other projects where even when llama.cpp is being used directly, it's being credited to Ollama: https://github.com/ggml-org/llama.cpp/pull/12896 Ollama doesn't contributed to upstream (that's fine, they're not obligated to), but it's a bit…
    comment by lhl · 2025-05-16 · in Ollama's new engine for multimodal models
  6. There's a pretty clear explanation of the llama.cpp history here: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd... I don't fully understand Ollama's timeline and strategy yet.
    comment by simonw · 2025-05-16 · in Ollama's new engine for multimodal models
  7. Because of things like this: https://github.com/ggml-org/llama.cpp/issues/12637 Where "supporting" a model doesn't mean what you think it means for cpp Between that and the long saga with vision models having only partial support, with a…
    comment by nkwaml · 2025-05-16 · in Ollama's new engine for multimodal models
  8. …this bug". A good example is this PR for llama.cpp which the author says was written 99% by AI. https://github.com/ggerganov/llama.cpp/pull/11453 When the problem statement can be narrowed down to "Here is some code that already produces a…
    comment by jwitthuhn · 2025-05-16 · in If AI is so good at coding where are the open source contributions?
  9. …in parallel (also, would you mind sharing a code pointer if you have any handy? I found this https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd... but not sure if that's the codepath taken)
    comment by brrrrrm · 2025-05-10 · in Vision Now Available in Llama.cpp
  10. …the same results - but I see on M1 with a 7b model we should expect ~10x faster prompt processing https://github.com/ggml-org/llama.cpp/discussions/4167 I wonder if it's the encoder that isn't optimized?
    comment by brrrrrm · 2025-05-10 · in Vision Now Available in Llama.cpp
  11. …already very decent output, describing different images pretty well. Steps to reproduce: git clone https://github.com/ggml-org/llama.cpp.git cmake -B build cmake --build build --config Release -j 12 --clean-first # download model and…
    comment by dust42 · 2025-05-10 · in Vision Now Available in Llama.cpp
  12. llama.cpp offers compiled releases for multiple platforms. This release has the new vision features: https://github.com/ggml-org/llama.cpp/releases/tag/b5332 On macOS I downloaded the llama-b5332-bin-macos-arm64.zip file and then had to…
    comment by simonw · 2025-05-10 · in Vision Now Available in Llama.cpp
  13. This is the most useful documentation I've found so far to help understand how this works: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd...
    comment by simonw · 2025-05-10 · in Vision Now Available in Llama.cpp
  14. ▲ 550
    Vision Now Available in Llama.cpp
    story by redman25 · 2025-05-10 · 104 comments
  15. …is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1] [0]: https://github.com/ggml-org/llama.cpp/issues/10453 [1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-184...
    comment by behnamoh · 2025-05-03 · in Run LLMs on Apple Neural Engine (ANE)
  16. …of programming will look quite different. In ten to fifteen years it will be unrecognizable. 1: https://github.com/ggml-org/llama.cpp/pull/11453
    comment by CamperBob2 · 2025-05-01 · in Two publishers and three authors fail to understand what "vibe coding" means
  17. …[2] unsloth quant: https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF [3] llama.cpp: https://github.com/ggml-org/llama.cpp
    comment by dust42 · 2025-05-01 · in Running Qwen3 on your macbook, using MLX, to vibe code for free
  18. …include in ollama's ModelFile are things you would pass to llama.cpp using command line flags: https://github.com/ggml-org/llama.cpp/blob/master/examples/m... If you only ever have one set of configuration parameters per model (same temp,…
    comment by rahimnathwani · 2025-04-30 · in Xiaomi MiMo Reasoning Model
  19. …Cuda / Metal for the bleeding edge models if they need a new operator. Try it out! Benchmarks here: https://github.com/ggml-org/llama.cpp/discussions/10879 (ollama doesn’t support vulkan for some weird reason. I guess they never pulled the…
    comment by pja · 2025-04-24 · in AMD Publishes Open-Source Driver for GPU Virtualization, Radeon "In the Roadmap"
  20. …4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889 3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was…
    comment by danielhanchen · 2025-04-24 · in Llama 4 Smells Bad
  21. This is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637 . This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a…
    comment by tbocek · 2025-04-20 · in Gemma 3 QAT Models: Bringing AI to Consumer GPUs
  22. …(Note that all comparisons with `llama.cpp` were made before I introduced the types from https://github.com/ggml-org/llama.cpp/pull/8151 where the 1.6-bit type uses the techniques described in the aforementioned blog post). I wanted to try…
    comment by compilade · 2025-04-17 · in Microsoft researchers developed a hyper-efficient AI model that can run on CPUs
  23. …> The llamafile executable size is increased from 30mb to 200mb by this release. This is caused by https://github.com/ggml-org/llama.cpp/issues/7156 . We're already employing some workarounds to minimize the impact of upstream development…
    comment by chrismorgan · 2025-04-07 · in Show HN: LocalScore – Local LLM Benchmark
  24. …the model format. - Both of these combined is what llama.cpp does, via @ochafik, in inter alia, https://github.com/ggml-org/llama.cpp/pull/9639 . - ollama isn't plugged into this system AFAIK To OP's question, specifying a format in the…
    comment by refulgentis · 2025-03-27 · in Gemma3 Function Calling
  25. I tried to execute the PoC by running the following: git clone https://github.com/ggml-org/llama.cpp.git && cd llama.cpp git checkout c0d4843225eed38903ea71ef302a02fa0b27f048 # Checkout a revision prior to the exploit fix in…
    comment by yamrzou · 2025-03-24 · in Heap-overflowing Llama.cpp to RCE
  26. …another handcrafted Siglip2 NN process. Interesting—they just figured out how to handle Siglip1. https://github.com/ggml-org/llama.cpp/pull/12344/commits/631...
    comment by changtimwu · 2025-03-13 · in Gemma 3 Technical Report [pdf]
  27. …single-socket benchmarks, and includes some optimizations. Just thought it was interesting. [1]: https://github.com/ggml-org/llama.cpp/discussions/11733
    comment by magicalhippo · 2025-03-10 · in Llama.cpp AI Performance with the GeForce RTX 5090 Review
  28. The implementation's here: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/g...
    comment by regularfry · 2025-03-10 · in Llama.cpp AI Performance with the GeForce RTX 5090 Review
  29. Looks like he bases the benchmarks off of https://github.com/ggml-org/llama.cpp/discussions/4167 I get why he calls it a simulator, as it can simulate token output. It's an important aspect for evaluating use case if you need to get a…
    comment by FloatArtifact · 2025-03-06 · in Apple unveils new Mac Studio
  30. …and DDR5 might make a difference in both performance and cost , Gen 4 or Gen 5 have 8 or 9 t/s https://github.com/ggml-org/llama.cpp/discussions/11733 ) 2. AMD Epyc + 4090 or 5090 running KTransformers (over 10 t/s decode ?…
    comment by wolfgangK · 2025-03-06 · in Apple M3 Ultra
  31. Googling for what you ask has found immediately this discussion: https://github.com/ggml-org/llama.cpp/discussions/11733 about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems. While it was rather tricky, after many…
    comment by adrian_b · 2025-03-05 · in Apple M3 Ultra
  32. …with bf16 and fp8, and the latter was only 2 Elo points below, well within measurement error. [0]: https://github.com/ggml-org/llama.cpp/blob/master/examples/q... [1]: https://github.com/ggml-org/llama.cpp/discussions/5063#discu... [2]:…
    comment by espadrine · 2025-03-05 · in Apple unveils new Mac Studio
  33. x2 speed increase for ggml by optimizing SIMD: https://github.com/ggml-org/llama.cpp/pull/11453 "99% written by DeepSeek-R1" according to the author.
    comment by CamperBob2 · 2025-02-24 · in DeepSeek Open Source FlashMLA – MLA Decoding Kernel for Hopper GPUs
  34. …Pi people don't care, lol. [1] https://github.com/deepseek-ai/DeepSeek-R1#distilled-model-e... [2] https://github.com/ggerganov/llama.cpp/pull/9639
    comment by tucnak · 2025-02-15 · in Deepseek R1 Distill 8B Q40 on 4 x Raspberry Pi 5
  35. …You could mimic either. [1] https://github.com/zed-industries/zed/tree/be830742439f531e8... [2] https://github.com/ggerganov/llama.cpp/blob/master/examples/... [3] https://github.com/zed-industries/zed/tree/be830742439f531e8...
    comment by spmurrayzzz · 2025-02-14 · in Zed now predicts your next edit with Zeta, our new open model
  36. …https://nitter.poast.org/ggerganov/status/188461277009384272... The same on Nvidia (various models) https://github.com/ggerganov/llama.cpp/issues/11474 [1] this is a the model: https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/De...
    comment by un_ess · 2025-02-11 · in Building a personal, private AI computer on a budget
  37. …But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage o
    comment by kamranjon · 2025-02-11 · in Building a personal, private AI computer on a budget
  38. …An M4 Pro still has only 273GB/s, while even the 2 generations old RTX 3090 has 935GB/s. https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by sofixa · 2025-02-11 · in Building a personal, private AI computer on a budget
  39. …to get good numbers. The theoretical limit should be higher, but it's not yet. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/11474#issuecom...
    comment by deoxykev · 2025-02-03 · in How to Run DeepSeek R1 671B Locally on a $2000 EPYC Server
  40. …a grounded idea of what models can handle. Mainly - tool calling support just merged in llama.cpp ( https://github.com/ggerganov/llama.cpp/pull/9639 ) this week, and it's been a fun exercise to put local LLMs through the wringer to see how…
    comment by horsawlarway · 2025-02-03 · in Goose: An open-source, extensible AI agent that goes beyond code suggestions
  41. ▲ 3
    Llama.cpp now supports tool calling (OpenAI-compatible)
    story by ochafik · 2025-02-01 · 1 comment
  42. …template library ( https://github.com/google/minja ) just to add tool call support to llama.cpp ( https://github.com/ggerganov/llama.cpp/pull/9639 ). (*): I mean, technically my employer's :-D*
    comment by ochafik · 2025-02-01 · in Ollama are 'try[ing to] achieve vendor lock-in'
  43. …they use code from other repositories that permit such a thing is tilting at windmills, IMHO. [0] https://github.com/ggerganov/llama.cpp/blob/master/LICENSE [1] https://en.wikipedia.org/wiki/MIT_License [2]…
    comment by davely · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  44. It's in one of the collapsed comments. Quoted below: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme... So Ollama are basically forking a little bit of everything to try and achieve vendor lock-in. Some examples: The Ollama…
    comment by lioeters · 2025-01-31 · in Ollama are 'try[ing to] achieve vendor lock-in'
  45. …same atm. Not sure about the plans for RamaLama, but it might be something worth to consider. [0] https://github.com/ggerganov/llama.cpp/issues/7252
    comment by ggerganov · 2025-01-31 · in RamaLama
  46. ▲ 17
    Ollama are 'try[ing to] achieve vendor lock-in'
    story by alexmorley · 2025-01-31 · 5 comments
  47. …work, it's just not the kind of work that hackers who hang out on forums like this appreciate. [0] https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  48. This is the point of it: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
    comment by eigenvalue · 2025-01-31 · in RamaLama
  49. …and others. Is that inaccurate, or something different? [edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8010
    comment by Gracana · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  50. …llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp
    comment by woadwarrior01 · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?