Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …limited you should go beyond 100 tokens/sec with the same model and a 8 bit quantization. 1. https://github.com/ggml-org/llama.cpp/discussions/15013
    comment by mysteria · 2025-09-18 · in Nvidia buys $5B in Intel
  2. From https://github.com/ggml-org/llama.cpp/issues/15940#issuecomm... : > This is a massive task, likely 2-3 months of full-time work for a highly specialized engineer. Until the Qwen team contributes the implementation, there are no quick…
    comment by NiekvdMaas · 2025-09-13 · in Qwen 3 now supports ARM and MLX
  3. https://github.com/ggml-org/llama.cpp/issues/15940
    comment by gre · 2025-09-12 · in Qwen3-Next
  4. This is the command probably: sudo sysctl iogpu.wired_limit_mb=184320 Source: https://github.com/ggml-org/llama.cpp/discussions/15396
    comment by frontsideair · 2025-09-09 · in Experimenting with Local LLMs on macOS
  5. …with both LPDDR5 and 32GB, only 16GB or LPDDR4(X). Using llama-bench, and Llama 2 7B Q4_0 like https://github.com/ggml-org/llama.cpp/discussions/10879 how does yours compare? Cuz I'm also comparing it with a few a few Ryzen 5 3000 Series…
    comment by ThatPlayer · 2025-09-08 · in Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
  6. …update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build…
    comment by danielhanchen · 2025-08-24 · in DeepSeek-v3.1
  7. …grammar, the grammar constraints are applied to the full vocabulary and the token is resampled." https://github.com/ggml-org/llama.cpp/blob/54a241f505d515d62...
    comment by refulgentis · 2025-08-22 · in DeepSeek-v3.1
  8. and in fact leverages that control to constrain outputs to those matching user-specified BNFs https://github.com/ggml-org/llama.cpp/tree/master/grammars
    comment by kristjansson · 2025-08-11 · in Ollama and gguf
  9. llama.cpp still doesn't support gpt-oss tool calling. https://github.com/ggml-org/llama.cpp/pull/15158 (among other similar PRs) But I also couldn't get vllm, or transformers serve, or ollama (400 response on /v1/chat/completions) working…
    comment by mhitza · 2025-08-11 · in Mistral Integration Improved in Llama.cpp
  10. It is fixed in this PR/branch: https://github.com/ggml-org/llama.cpp/pull/15181
    comment by tarruda · 2025-08-11 · in GPT-OSS-120B runs on just 8GB VRAM & 64GB+ system RAM
  11. …tokens for an AI model that's running on an Intel Arc, Nvidia 3090, and AMD gpu at the same time. https://github.com/ggml-org/llama.cpp/pull/5321
    comment by jychang · 2025-08-10 · in The Framework Desktop is a beast
  12. …Implementation for gpt-oss this week showed 2-3x improvements https://github.com/ggml-org/llama.cpp/pull/15157 https://www.reddit.com/r/LocalLLaMA/comments/1mkowrw/llamacp...
    comment by Maxious · 2025-08-09 · in How attention sinks keep language models stable
  13. And, as always, the FOSS ecosystem moves quickly, llama.cpp already fully support them! https://github.com/ggml-org/llama.cpp/pull/15157
    comment by diggan · 2025-08-08 · in How attention sinks keep language models stable
  14. Holy smokes, there's already llama.cpp support: https://github.com/ggml-org/llama.cpp/pull/15091
    comment by HanClinto · 2025-08-05 · in Open models by OpenAI
  15. …are we now? But not seeing any evidence that there's popular support in any of the main frameworks. https://github.com/ggml-org/llama.cpp/issues/1499 https://github.com/ollama/ollama/issues/5186 Good news, AMD has an initial implementation…
    comment by jauntywundrkind · 2025-07-30 · in My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
  16. …incorrectly, since chatting with the model will use the actual correct token. Another is Llama 4 - https://github.com/ggml-org/llama.cpp/pull/12889 in which I fixed a RoPE issue - if we didn't fix it first, then again the calibration…
    comment by danielhanchen · 2025-07-23 · in Qwen3-Coder: Agentic coding in the world
  17. …they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3 to MLX-LM as well: You can run it via `mlx_lm.chat --model…
    comment by reach-vb · 2025-07-09 · in Smollm3: Smol, multilingual, long-context reasoner LLM
  18. The average consumer uses llama.cpp. So here is your list of kernels: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/g... And here is pretty damning evidence that you're full of shit: https://github.com/ggml-org/llama.cpp/blob/m…
    comment by imtringued · 2025-06-16 · in AMD's AI Future Is Rack Scale 'Helios'
  19. Inference Library: An LLM inference library built over https://github.com/ggerganov/llama.cpp library for integrating LLMs into programs. Model Context Protocol Library: A C++ MCP client/server library that includes all fundamental…
    comment by xentnex · 2025-06-13 · in C++ MCP Client/Server SDK
  20. …applications. Here is a list of libraries: Inference Library: An LLM inference library built over https://github.com/ggerganov/llama.cpp library for integrating LLMs into programs. Model Context Protocol Library: An MCP client/server…
    comment by npalli · 2025-06-12 · in Mbase, an LLM SDK in C++
  21. > for near negligible drop in quality Hmm, that's evidently and anecdotally wrong: https://github.com/ggml-org/llama.cpp/discussions/4110
    comment by behnamoh · 2025-06-10 · in OpenAI dropped the price of o3 by 80%
  22. …power saving mode, inside ~8W. Come to think of it, those results might make a good blog post. [0] https://github.com/ggml-org/llama.cpp/discussions/10879 Search for "HX 370"
    comment by jakogut · 2025-06-06 · in AMD Radeon 8050S “Strix Halo” Linux Graphics Performance Review
  23. …of doing that, so I don't know how well the increased parameter size balances the quantization. See https://github.com/ggml-org/llama.cpp/pull/1684 and https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... for comparisons between…
    comment by rhdunn · 2025-05-30 · in Ask HN: What is the best LLM for consumer grade hardware?
  24. …started and probably all you need. I use it a lot on my M2 Macbook Air and it's really handy. [0] - https://github.com/ggml-org/llama.cpp [1] - https://lmstudio.ai/
    comment by thatcherc · 2025-05-21 · in Devstral
  25. …because this functionality has been in llama.cpp since 2023 if I understand this PR correctly: https://github.com/ggml-org/llama.cpp/pull/4312 Instead of offering a forked llama.cpp with the changes applied as commits, the repo wants you…
    comment by Aurornis · 2025-05-17 · in Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon
  26. > Citation? https://github.com/ollama/ollama/issues/7300#issuecomment-24... https://github.com/ggml-org/llama.cpp/blob/3e0be1cacef290c99... Anyway my point was just that it's not as easy as just pushing a patch upstream, like it is in many…
    comment by magicalhippo · 2025-05-17 · in Ollama's new engine for multimodal models
  27. “The llama.cpp project is the main playground for developing new features for the ggml library” -- https://github.com/ggml-org/llama.cpp “Some of the development is currently happening in the llama.cpp and whisper.cpp repos” --…
    comment by cwillu · 2025-05-16 · in Ollama's new engine for multimodal models
  28. FWIW, llama.cpp links to and fetches models from ollama ( https://github.com/ggml-org/llama.cpp/blob/master/tools/run/... ). This issue seems to be the typical case of someone being bothered for someone else, because it implies there's no…
    comment by levifig · 2025-05-16 · in Ollama violating llama.cpp license for over a year
  29. …llama.cpp's LICENSE: "Copyright (c) 2023-2024 The ggml authors". ollama doesn't include it. [0]: https://github.com/ggml-org/llama.cpp/blob/master/LICENSE#L3...
    comment by camgunz · 2025-05-16 · in Ollama violating llama.cpp license for over a year
  30. …license is in the ollama project as is in the llama.cpp project, is this not sufficient? llamma.cpp https://github.com/ggml-org/llama.cpp/blob/master/LICENSE ollama.cpp https://github.com/ollama/ollama/blob/main/LICENSE
    comment by gmm1990 · 2025-05-16 · in Ollama violating llama.cpp license for over a year
  31. llama.cpp clearly does not support iSWA: https://github.com/ggml-org/llama.cpp/issues/12637 Ollama does, please try it.
    comment by nolist_policy · 2025-05-16 · in Ollama's new engine for multimodal models
  32. …is written in golang so of course they can not meaningfully contribute that back to llama.cpp. [0] https://github.com/ggml-org/llama.cpp/issues/12637
    comment by nolist_policy · 2025-05-16 · in Ollama's new engine for multimodal models
  33. …other projects where even when llama.cpp is being used directly, it's being credited to Ollama: https://github.com/ggml-org/llama.cpp/pull/12896 Ollama doesn't contributed to upstream (that's fine, they're not obligated to), but it's a bit…
    comment by lhl · 2025-05-16 · in Ollama's new engine for multimodal models
  34. There's a pretty clear explanation of the llama.cpp history here: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd... I don't fully understand Ollama's timeline and strategy yet.
    comment by simonw · 2025-05-16 · in Ollama's new engine for multimodal models
  35. Because of things like this: https://github.com/ggml-org/llama.cpp/issues/12637 Where "supporting" a model doesn't mean what you think it means for cpp Between that and the long saga with vision models having only partial support, with a…
    comment by nkwaml · 2025-05-16 · in Ollama's new engine for multimodal models
  36. …this bug". A good example is this PR for llama.cpp which the author says was written 99% by AI. https://github.com/ggerganov/llama.cpp/pull/11453 When the problem statement can be narrowed down to "Here is some code that already produces a…
    comment by jwitthuhn · 2025-05-16 · in If AI is so good at coding where are the open source contributions?
  37. …in parallel (also, would you mind sharing a code pointer if you have any handy? I found this https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd... but not sure if that's the codepath taken)
    comment by brrrrrm · 2025-05-10 · in Vision Now Available in Llama.cpp
  38. …the same results - but I see on M1 with a 7b model we should expect ~10x faster prompt processing https://github.com/ggml-org/llama.cpp/discussions/4167 I wonder if it's the encoder that isn't optimized?
    comment by brrrrrm · 2025-05-10 · in Vision Now Available in Llama.cpp
  39. …already very decent output, describing different images pretty well. Steps to reproduce: git clone https://github.com/ggml-org/llama.cpp.git cmake -B build cmake --build build --config Release -j 12 --clean-first # download model and…
    comment by dust42 · 2025-05-10 · in Vision Now Available in Llama.cpp
  40. llama.cpp offers compiled releases for multiple platforms. This release has the new vision features: https://github.com/ggml-org/llama.cpp/releases/tag/b5332 On macOS I downloaded the llama-b5332-bin-macos-arm64.zip file and then had to…
    comment by simonw · 2025-05-10 · in Vision Now Available in Llama.cpp
  41. This is the most useful documentation I've found so far to help understand how this works: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd...
    comment by simonw · 2025-05-10 · in Vision Now Available in Llama.cpp
  42. …is that MLX is made by Apple and yet, they can't support ANE given its closed-source API! [1] [0]: https://github.com/ggml-org/llama.cpp/issues/10453 [1]: https://github.com/ml-explore/mlx/issues/18#issuecomment-184...
    comment by behnamoh · 2025-05-03 · in Run LLMs on Apple Neural Engine (ANE)
  43. …of programming will look quite different. In ten to fifteen years it will be unrecognizable. 1: https://github.com/ggml-org/llama.cpp/pull/11453
    comment by CamperBob2 · 2025-05-01 · in Two publishers and three authors fail to understand what "vibe coding" means
  44. …[2] unsloth quant: https://huggingface.co/unsloth/Qwen3-30B-A3B-GGUF [3] llama.cpp: https://github.com/ggml-org/llama.cpp
    comment by dust42 · 2025-05-01 · in Running Qwen3 on your macbook, using MLX, to vibe code for free
  45. …include in ollama's ModelFile are things you would pass to llama.cpp using command line flags: https://github.com/ggml-org/llama.cpp/blob/master/examples/m... If you only ever have one set of configuration parameters per model (same temp,…
    comment by rahimnathwani · 2025-04-30 · in Xiaomi MiMo Reasoning Model
  46. …Cuda / Metal for the bleeding edge models if they need a new operator. Try it out! Benchmarks here: https://github.com/ggml-org/llama.cpp/discussions/10879 (ollama doesn’t support vulkan for some weird reason. I guess they never pulled the…
    comment by pja · 2025-04-24 · in AMD Publishes Open-Source Driver for GPU Virtualization, Radeon "In the Roadmap"
  47. …4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889 3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was…
    comment by danielhanchen · 2025-04-24 · in Llama 4 Smells Bad
  48. This is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637 . This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a…
    comment by tbocek · 2025-04-20 · in Gemma 3 QAT Models: Bringing AI to Consumer GPUs
  49. …(Note that all comparisons with `llama.cpp` were made before I introduced the types from https://github.com/ggml-org/llama.cpp/pull/8151 where the 1.6-bit type uses the techniques described in the aforementioned blog post). I wanted to try…
    comment by compilade · 2025-04-17 · in Microsoft researchers developed a hyper-efficient AI model that can run on CPUs
  50. …> The llamafile executable size is increased from 30mb to 200mb by this release. This is caused by https://github.com/ggml-org/llama.cpp/issues/7156 . We're already employing some workarounds to minimize the impact of upstream development…
    comment by chrismorgan · 2025-04-07 · in Show HN: LocalScore – Local LLM Benchmark