Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …:) [0] https://www.reddit.com/r/LocalLLaMA/ [1] https://lmstudio.ai/ [2] https://ollama.com/ [3] https://github.com/ggml-org/llama.cpp [4] https://huggingface.co/unsloth/collections
    comment by lgl · 2026-06-22 · in Ask HN: How close are we to local LLMs being useful? What's the impact?
  2. …it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730
    comment by xrd · 2026-06-22 · in GLM 5.2 vs. Opus
  3. https://github.com/ggml-org/llama.cpp/pull/24219
    comment by trykhlieb · 2026-06-21 · in Show HN: VLMs Can Respond Twice as Fast Without Losing Quality
  4. …figured it should make on its own. [1]: https://sleepingrobots.com/dreams/stop-using-ollama/ [2]: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... [3]: https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909...
    comment by markussss · 2026-06-17 · in Running local models is good now
  5. …decoding functionality [0] as it allows me to iterate on the implementation much faster. [0] https://github.com/ggml-org/llama.cpp/pull/19164
    comment by ggerganov · 2026-06-16 · in Running local models is good now
  6. …run it on the Mac too. [0] - https://github.com/search?q=%22Assisted-by%22+user%3Aggml-or... [1] - https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYS...
    comment by ggerganov · 2026-06-16 · in Running local models is good now
  7. …that will sweep the arguments for you so you don't have to restart the server and send it prompts: https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... EDIT: Also the section about downloading the models should have mentioned…
    comment by Aurornis · 2026-06-12 · in How to setup a local coding agent on macOS
  8. …README in the Unsloth list of files: llama.cpp is working on a PR to support the gemma4 drafters: https://github.com/ggml-org/llama.cpp/pull/23398 . Also note the PR submitter didn't experience much speedup with 26B (seems typical that MoE…
    comment by pfheatwole · 2026-06-05 · in Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
  9. I just build llama.cpp from scratch on the PR that has MTP drafters. https://github.com/ggml-org/llama.cpp/pull/23398 Please don't use Ollama, it's a bad actor in the OSS community.
    comment by thot_experiment · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  10. I think MTP Gemma4 support is still WIP https://github.com/ggml-org/llama.cpp/pull/23398 ?
    comment by kristjansson · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  11. …prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746
    comment by verdverm · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  12. …plain llama.cpp it looks like a PR has already merged for this model with vision and audio support: https://github.com/ggml-org/llama.cpp/pull/24077
    comment by philipkglass · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  13. llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples... When comparing…
    comment by Aurornis · 2026-06-01 · in A 10 year old Xeon is all you need
  14. …on llama.cpp; similar work for Qwen3.6 landed recently and has been great thus far. [0]: https://github.com/ggml-org/llama.cpp/pull/23398
    comment by kristjansson · 2026-06-01 · in A 10 year old Xeon is all you need
  15. …all of the reduced precision fp8, fp4 types today: by storing them in the corresponding uint: https://github.com/ggml-org/llama.cpp/discussions/15095
    comment by mathisfun123 · 2026-05-31 · in Racket v9.2
  16. https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... llama.cpp supports grammar limiting using either GBNF or json schema (It just translate it to GBNF behind the scenes I think). So I have my harness generate a tool schema on…
    comment by peer0 · 2026-05-20 · in Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
  17. because llama.cpp doesn't accept fully pr made by ai agents even if they are guided by the author https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md
    comment by skiwithuge · 2026-05-15 · in A few words on DS4
  18. …https://github.com/antirez/ds4/issues/108 Guess I'll have to keep watching the llama.cpp issue[1] [1] https://github.com/ggml-org/llama.cpp/issues/22319
    comment by karmakaze · 2026-05-15 · in A few words on DS4
  19. …last cmake command for parallel build) and llama-server with --reasoning off Note that the MTP PR https://github.com/ggml-org/llama.cpp/pull/22673 is still under development, so things might be broken.
    comment by johndough · 2026-05-12 · in If AI writes your code, why use Python?
  20. …compiled on Windows before use. The GitHib project has a cornucopia of pre-built artifacts to use. https://github.com/ggml-org/llama.cpp/releases
    comment by verytrivial · 2026-05-07 · in Diskless Linux boot using ZFS, iSCSI and PXE
  21. There's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978 , I can't wait to see how it will compare against MTP
    comment by egeres · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  22. There is a newer PR which will probably be merged soon: https://github.com/ggml-org/llama.cpp/pull/22673
    comment by tarruda · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  23. MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed…
    comment by zdw · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  24. …(or llama.cpp[2]), so it may take a while. [1] https://github.com/ml-explore/mlx-lm/pull/990 [2] https://github.com/ggml-org/llama.cpp/pull/22673
    comment by dvt · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  25. the build they use is from February, over two months old: https://github.com/ggml-org/llama.cpp/releases/tag/b8121 Which might not sound like much, but 2months in llm time is a long time, especially regarding support for new hardware like…
    comment by andy_xor_andrew · 2026-04-28 · in Intel Arc Pro B70 Review
  26. …that is being actively developed and is not that far behind CUDA where speed is concerned. * https://github.com/ggml-org/llama.cpp/releases
    comment by Eisenstein · 2026-04-24 · in DeepSeek v4
  27. …will get you close to like ~Sonnet 4 though: Grab a recent win-vulkan-x64 build of llama.cpp here: https://github.com/ggml-org/llama.cpp/releases - llama.cpp is the engine used by Ollama and common wisdom is to just use it directly. You…
    comment by daemonologist · 2026-04-21 · in Qwen3.6-Max-Preview: Smarter, Sharper, Still Evolving
  28. …llama.cpp that would be much more useful for experimenting. One example I picked at random: https://github.com/ggml-org/llama.cpp/pull/22105
    comment by Aurornis · 2026-04-20 · in We got 207 tok/s with Qwen3.5-27B on an RTX 3090
  29. …The llama.cpp folks are still thinking about it, per a recent discussion from that project: https://github.com/ggml-org/llama.cpp/discussions/21961
    comment by zozbot234 · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  30. …CUDA specific. It made a pretty significant difference (> 40% tg/s) on my Ryzen 7840U laptop. 1 - https://github.com/ggml-org/llama.cpp/blob/master/docs/build...
    comment by rubiquity · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  31. …are fixed. The 4th is Google themselves improving the chat template for tool calling for Gemma. https://github.com/ggml-org/llama.cpp/issues/21255 was another issue CUDA 13.2 was broken - this was NVIDIA's CUDA compiler itself breaking -…
    comment by danielhanchen · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  32. …better than it used to be and honestly way more powerful. If you haven't looked lately, you should https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
    comment by ottah · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  33. …from it internally for the KV cache [0] So a quantized KV cache now must see less degradation [0] https://github.com/ggml-org/llama.cpp/pull/21038
    comment by kgeist · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  34. check on same port, there is an OpenAI API https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
    comment by homarp · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  35. …8000 (with MCP support and web chat interface) and you have OpenAI API on the same 8000 port. ( https://github.com/ggml-org/llama.cpp/tree/master/tools/serv... lists the endpoints)
    comment by homarp · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  36. Model authors are welcome to add support to llama.cpp before release like IBM did for granite 4 https://github.com/ggml-org/llama.cpp/pull/13550
    comment by Maxious · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  37. You can have multiple models served now with loading/unloading with just the server binary. https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by dminik · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  38. https://github.com/ggml-org/llama.cpp
    comment by throwaway2027 · 2026-04-15 · in Elevated errors on Claude.ai, API, Claude Code
  39. …). 2) Use the older completions endpoint (llama.cpp's responses support is incomplete - https://github.com/ggml-org/llama.cpp/issues/19138 )
    comment by vsrinivas · 2026-04-13 · in I ran Gemma 4 as a local model in Codex CLI
  40. …https://unsloth.ai/docs/models/gemma-4 https://unsloth.ai/docs/models/qwen3.5 https://github.com/ggml-org/llama.cpp
    comment by throwaway2027 · 2026-04-12 · in Pro Max 5x quota exhausted in 1.5 hours despite moderate usage
  41. …making important choices, while making generation of the "obvious" tokens cheaper and faster. [0] https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... [1] https://developers.redhat.com/articles/2025/06/03/structured...
    comment by vova_hn2 · 2026-04-04 · in Embarrassingly simple self-distillation improves code generation
  42. …says it was generated with Claude Code so it has a very low chance of getting merged upstream. [0] https://github.com/ggml-org/llama.cpp/pull/21089 [1] https://github.com/ggml-org/llama.cpp/pulls?q=Turboquant+is%...
    comment by kgeist · 2026-04-04
  43. …Unless you want CUDA support on Linux I guess, dunno why you'd need such a silly thing though: https://github.com/ggml-org/llama.cpp/releases
    comment by dTal · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini
  44. I googled it- supposed fixed template https://github.com/ggml-org/llama.cpp/issues/21347#issuecomm...
    comment by jasonjmcghee · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini
  45. That also comes upstream from llama.cpp https://github.com/ggml-org/llama.cpp/discussions/4345
    comment by wronglebowski · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini
  46. …instead of JSON's double quotes, and disables json-to-schema conversion for these types." [1]: https://github.com/ggml-org/llama.cpp/pull/21326/commits/a50... edit: formatting
    comment by airspresso · 2026-04-03 · in Google releases Gemma 4 open models
  47. …an nVidia 16GB GPU, but Ollama's a bit easier to ease into since they have a preset model library. https://github.com/ggml-org/llama.cpp
    comment by vunderba · 2026-04-02 · in Google releases Gemma 4 open models
  48. …but even then barely enough for 64K for 24GB VRAM. Maybe 8 bit KV quantization is fine now after https://github.com/ggml-org/llama.cpp/pull/21038 got merged, so 100K+ is possible. > I should pick a full precision smaller model or 4 bit…
    comment by petu · 2026-04-02 · in Google releases Gemma 4 open models
  49. …with ROCm and Vulkan. These are sourced from https://github.com/lemonade-sdk/llamacpp-rocm and https://github.com/ggml-org/llama.cpp/releases respectively.
    comment by sawansri · 2026-04-02 · in Lemonade by AMD: a fast and open source local LLM server using GPU and NPU
  50. …judge open models on launch day... it never goes well. There are ~always bugs on launch day. [0]: https://github.com/ggml-org/llama.cpp/pull/21326 [1]: https://github.com/ggml-org/llama.cpp/issues/21316
    comment by coder543 · 2026-04-02 · in Google releases Gemma 4 open models