Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. wonder if theyre basically doing this llamacpp reasoning trick: https://github.com/ggml-org/llama.cpp/blob/master/common/rea... id guess its the harness
    comment by cyanydeez · 2026-07-01 · in Codex reasoning-token clustering at 516 may be leading to degraded performance
  2. ▲ 3
    TurboPrefill: 2.7× faster than llama.cpp Pipeline Parallel on Llama-3-70B
    story by trykhlieb · 2026-06-30 · 0 comments
  3. …for characteristics and comparative performance of all M variants, if anybody is interested, is https://github.com/ggml-org/llama.cpp/discussions/4167?sort=... Its sister discussion for nvidia gpus is https://github.com/ggml-org/llama.cpp/d…
    comment by freehorse · 2026-06-29 · in Qwen 3.6 27B is the sweet spot for local development
  4. …+90% 5× RTX 5060 Ti: +90% 8× RTX 5060 Ti: +120% 10× P104-100 Pascal: +350% RFC / PoC discussion: https://github.com/ggml-org/llama.cpp/pull/24219
    comment by trykhlieb · 2026-06-22 · in Show HN: VLMs Can Respond Twice as Fast Without Losing Quality
  5. …:) [0] https://www.reddit.com/r/LocalLLaMA/ [1] https://lmstudio.ai/ [2] https://ollama.com/ [3] https://github.com/ggml-org/llama.cpp [4] https://huggingface.co/unsloth/collections
    comment by lgl · 2026-06-22 · in Ask HN: How close are we to local LLMs being useful? What's the impact?
  6. …it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730
    comment by xrd · 2026-06-22 · in GLM 5.2 vs. Opus
  7. https://github.com/ggml-org/llama.cpp/pull/24219
    comment by trykhlieb · 2026-06-21 · in Show HN: VLMs Can Respond Twice as Fast Without Losing Quality
  8. …figured it should make on its own. [1]: https://sleepingrobots.com/dreams/stop-using-ollama/ [2]: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... [3]: https://gist.github.com/jscott3201/e4b155885cc68c038d6ac8909...
    comment by markussss · 2026-06-17 · in Running local models is good now
  9. …decoding functionality [0] as it allows me to iterate on the implementation much faster. [0] https://github.com/ggml-org/llama.cpp/pull/19164
    comment by ggerganov · 2026-06-16 · in Running local models is good now
  10. …run it on the Mac too. [0] - https://github.com/search?q=%22Assisted-by%22+user%3Aggml-or... [1] - https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYS...
    comment by ggerganov · 2026-06-16 · in Running local models is good now
  11. …that will sweep the arguments for you so you don't have to restart the server and send it prompts: https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... EDIT: Also the section about downloading the models should have mentioned…
    comment by Aurornis · 2026-06-12 · in How to setup a local coding agent on macOS
  12. …README in the Unsloth list of files: llama.cpp is working on a PR to support the gemma4 drafters: https://github.com/ggml-org/llama.cpp/pull/23398 . Also note the PR submitter didn't experience much speedup with 26B (seems typical that MoE…
    comment by pfheatwole · 2026-06-05 · in Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
  13. I just build llama.cpp from scratch on the PR that has MTP drafters. https://github.com/ggml-org/llama.cpp/pull/23398 Please don't use Ollama, it's a bad actor in the OSS community.
    comment by thot_experiment · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  14. I think MTP Gemma4 support is still WIP https://github.com/ggml-org/llama.cpp/pull/23398 ?
    comment by kristjansson · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  15. …prompt cache bugs in qwen/gemma models This is a good starting issue with a bunch of linked/related https://github.com/ggml-org/llama.cpp/issues/22746
    comment by verdverm · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  16. …plain llama.cpp it looks like a PR has already merged for this model with vision and audio support: https://github.com/ggml-org/llama.cpp/pull/24077
    comment by philipkglass · 2026-06-03 · in Gemma 4 12B: A unified, encoder-free multimodal model
  17. llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples... When comparing…
    comment by Aurornis · 2026-06-01 · in A 10 year old Xeon is all you need
  18. …on llama.cpp; similar work for Qwen3.6 landed recently and has been great thus far. [0]: https://github.com/ggml-org/llama.cpp/pull/23398
    comment by kristjansson · 2026-06-01 · in A 10 year old Xeon is all you need
  19. …all of the reduced precision fp8, fp4 types today: by storing them in the corresponding uint: https://github.com/ggml-org/llama.cpp/discussions/15095
    comment by mathisfun123 · 2026-05-31 · in Racket v9.2
  20. https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... llama.cpp supports grammar limiting using either GBNF or json schema (It just translate it to GBNF behind the scenes I think). So I have my harness generate a tool schema on…
    comment by peer0 · 2026-05-20 · in Show HN: Forge – Guardrails take an 8B model from 53% to 99% on agentic tasks
  21. ▲ 2
    Llama.cpp b9180: MTP support landed
    story by usagisushi · 2026-05-17 · 0 comments
  22. because llama.cpp doesn't accept fully pr made by ai agents even if they are guided by the author https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md
    comment by skiwithuge · 2026-05-15 · in A few words on DS4
  23. …https://github.com/antirez/ds4/issues/108 Guess I'll have to keep watching the llama.cpp issue[1] [1] https://github.com/ggml-org/llama.cpp/issues/22319
    comment by karmakaze · 2026-05-15 · in A few words on DS4
  24. …last cmake command for parallel build) and llama-server with --reasoning off Note that the MTP PR https://github.com/ggml-org/llama.cpp/pull/22673 is still under development, so things might be broken.
    comment by johndough · 2026-05-12 · in If AI writes your code, why use Python?
  25. ▲ 3
    Grinder12: 0.96-Bit Lossless Streaming KV-Cache (16.55x VRAM Savings
    story by AMICLLC · 2026-05-10 · 0 comments
  26. ▲ 1
    Llama and Spec: MTP Support
    story by jhoho · 2026-05-07 · 0 comments
  27. …compiled on Windows before use. The GitHib project has a cornucopia of pre-built artifacts to use. https://github.com/ggml-org/llama.cpp/releases
    comment by verytrivial · 2026-05-07 · in Diskless Linux boot using ZFS, iSCSI and PXE
  28. There's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978 , I can't wait to see how it will compare against MTP
    comment by egeres · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  29. There is a newer PR which will probably be merged soon: https://github.com/ggml-org/llama.cpp/pull/22673
    comment by tarruda · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  30. MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed…
    comment by zdw · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  31. …(or llama.cpp[2]), so it may take a while. [1] https://github.com/ml-explore/mlx-lm/pull/990 [2] https://github.com/ggml-org/llama.cpp/pull/22673
    comment by dvt · 2026-05-05 · in Accelerating Gemma 4: faster inference with multi-token prediction drafters
  32. the build they use is from February, over two months old: https://github.com/ggml-org/llama.cpp/releases/tag/b8121 Which might not sound like much, but 2months in llm time is a long time, especially regarding support for new hardware like…
    comment by andy_xor_andrew · 2026-04-28 · in Intel Arc Pro B70 Review
  33. …that is being actively developed and is not that far behind CUDA where speed is concerned. * https://github.com/ggml-org/llama.cpp/releases
    comment by Eisenstein · 2026-04-24 · in DeepSeek v4
  34. …will get you close to like ~Sonnet 4 though: Grab a recent win-vulkan-x64 build of llama.cpp here: https://github.com/ggml-org/llama.cpp/releases - llama.cpp is the engine used by Ollama and common wisdom is to just use it directly. You…
    comment by daemonologist · 2026-04-21 · in Qwen3.6-Max-Preview: Smarter, Sharper, Still Evolving
  35. …llama.cpp that would be much more useful for experimenting. One example I picked at random: https://github.com/ggml-org/llama.cpp/pull/22105
    comment by Aurornis · 2026-04-20 · in We got 207 tok/s with Qwen3.5-27B on an RTX 3090
  36. …The llama.cpp folks are still thinking about it, per a recent discussion from that project: https://github.com/ggml-org/llama.cpp/discussions/21961
    comment by zozbot234 · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  37. …CUDA specific. It made a pretty significant difference (> 40% tg/s) on my Ryzen 7840U laptop. 1 - https://github.com/ggml-org/llama.cpp/blob/master/docs/build...
    comment by rubiquity · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  38. …are fixed. The 4th is Google themselves improving the chat template for tool calling for Gemma. https://github.com/ggml-org/llama.cpp/issues/21255 was another issue CUDA 13.2 was broken - this was NVIDIA's CUDA compiler itself breaking -…
    comment by danielhanchen · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  39. …better than it used to be and honestly way more powerful. If you haven't looked lately, you should https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
    comment by ottah · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  40. …from it internally for the KV cache [0] So a quantized KV cache now must see less degradation [0] https://github.com/ggml-org/llama.cpp/pull/21038
    comment by kgeist · 2026-04-16 · in Qwen3.6-35B-A3B: Agentic coding power, now open to all
  41. check on same port, there is an OpenAI API https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
    comment by homarp · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  42. …8000 (with MCP support and web chat interface) and you have OpenAI API on the same 8000 port. ( https://github.com/ggml-org/llama.cpp/tree/master/tools/serv... lists the endpoints)
    comment by homarp · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  43. Model authors are welcome to add support to llama.cpp before release like IBM did for granite 4 https://github.com/ggml-org/llama.cpp/pull/13550
    comment by Maxious · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  44. You can have multiple models served now with loading/unloading with just the server binary. https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by dminik · 2026-04-16 · in The local LLM ecosystem doesn’t need Ollama
  45. https://github.com/ggml-org/llama.cpp
    comment by throwaway2027 · 2026-04-15 · in Elevated errors on Claude.ai, API, Claude Code
  46. …). 2) Use the older completions endpoint (llama.cpp's responses support is incomplete - https://github.com/ggml-org/llama.cpp/issues/19138 )
    comment by vsrinivas · 2026-04-13 · in I ran Gemma 4 as a local model in Codex CLI
  47. …https://unsloth.ai/docs/models/gemma-4 https://unsloth.ai/docs/models/qwen3.5 https://github.com/ggml-org/llama.cpp
    comment by throwaway2027 · 2026-04-12 · in Pro Max 5x quota exhausted in 1.5 hours despite moderate usage
  48. …making important choices, while making generation of the "obvious" tokens cheaper and faster. [0] https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... [1] https://developers.redhat.com/articles/2025/06/03/structured...
    comment by vova_hn2 · 2026-04-04 · in Embarrassingly simple self-distillation improves code generation
  49. …says it was generated with Claude Code so it has a very low chance of getting merged upstream. [0] https://github.com/ggml-org/llama.cpp/pull/21089 [1] https://github.com/ggml-org/llama.cpp/pulls?q=Turboquant+is%...
    comment by kgeist · 2026-04-04
  50. …Unless you want CUDA support on Linux I guess, dunno why you'd need such a silly thing though: https://github.com/ggml-org/llama.cpp/releases
    comment by dTal · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini