Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …Rotation. This seems to have genuinely helped KV cache compression as per [llama.cpp latest tests]( https://github.com/ggml-org/llama.cpp/pull/21038 ). On the other hand, [ik_llama.cpp]( https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/…
    comment by 3abiton · 2026-04-02 · in Claude Code users hitting usage limits 'way faster than expected'
  2. …(at least for prefill, which is a key bottleneck on pre-M5 platforms) is also in the works, per https://github.com/ggml-org/llama.cpp/issues/10453#issuecomm...
    comment by zozbot234 · 2026-03-31 · in Ollama is now powered by MLX on Apple Silicon in preview
  3. …than this. Starting with the llama.cpp discussion can give some better info than this blog post: https://github.com/ggml-org/llama.cpp/discussions/20969
    comment by Aurornis · 2026-03-27 · in TurboQuant: Building a Sub-Byte KV Cache Quantizer from Paper to Production
  4. …really need to specify it as a CLI option.) OP has raised a discussion with the llama.cpp folks https://github.com/ggml-org/llama.cpp/discussions/20852 but little interest so far
    comment by zozbot234 · 2026-03-24 · in Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
  5. …being eminently public and applied across multiple relevant dimensions: raw inference speed ( https://github.com/ggml-org/llama.cpp/releases ), external-facing capabilities ( https://github.com/open-webui/open-webui/releases ) and…
    comment by vntok · 2026-03-23 · in iPhone 17 Pro Demonstrated Running a 400B LLM
  6. …days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp ( https://github.com/ggml-org/llama.cpp/pull/20361#issuecommen... ): % llama-bench -m ~/ml-models/huggingface/ubergarm/Qwen3.5-397B-A17B-GGUF/…
    comment by tarruda · 2026-03-22 · in Flash-MoE: Running a 397B Parameter Model on a Laptop
  7. …extensive support for doing this and controlling exactly which weights end up where (llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/... , vllm: https://docs.vllm.ai/en/stable/configuration/engine_args/#of... ,…
    comment by daemonologist · 2026-03-22 · in Flash-MoE: Running a 397B Parameter Model on a Laptop
  8. Speculative decoding[1]? [1]: https://github.com/ggml-org/llama.cpp/blob/master/docs/specu...
    comment by genpfault · 2026-03-14 · in Can I run AI locally?
  9. https://github.com/ggml-org/llama.cpp/releases - has mac binaries https://unsloth.ai/docs/models/qwen3.5 - running locally guide for the Qwen 3.5 family of models, which have a range of different sizes.
    comment by kristianp · 2026-03-14 · in Can I run AI locally?
  10. …and checking it out. Fyi there's someone else on github keeping track of some LLM benchmarks too: https://github.com/ggml-org/llama.cpp/discussions/4167
    comment by SkyMarshal · 2026-03-05 · in Show HN: Open dataset of real-world LLM performance on Apple Silicon
  11. …in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...]…
    comment by simonw · 2026-02-20 · in Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI
  12. This is the easiest set up on a Mac. You need at least 16gb on a MacBook: https://github.com/ggml-org/llama.cpp/discussions/15396
    comment by ontouchstart · 2026-02-20 · in Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI
  13. …It is a bug in the model weights and reproducible in their official chat UI. More details here: https://github.com/ggml-org/llama.cpp/pull/19283#issuecommen...
    comment by tarruda · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed
  14. …to generate the appropriate "I'm done thinking" indicator. You can see this described in this PR https://github.com/ggml-org/llama.cpp/pull/19635 Apparently Step 3.5 Flash uses an odd format for its tags so llama.cpp just doesn't handle it…
    comment by sosodev · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed
  15. …chains. - Current release has a bug where sometimes it goes into an infinite reasoning loop: https://github.com/ggml-org/llama.cpp/pull/19283#issuecommen... Hopefully StepFun will do a new release which addresses these issues. BTW StepFun…
    comment by tarruda · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed
  16. Just in case anyone hasn't seen this yet: https://github.com/ggml-org/llama.cpp/discussions/15396 a guide for running gpt-oss on llama-server, with settings for various amounts of GPU memory, from 8GB on up
    comment by Twirrim · 2026-02-15 · in IBM tripling entry-level jobs after finding the limits of AI adoption
  17. …[1], which is useful if you are working with a custom programming language for example. [1] https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
    comment by threeducks · 2026-02-05 · in Claude Opus 4.6
  18. Georgi's relevant comment: https://github.com/ggml-org/llama.cpp/pull/19324#issuecommen...
    comment by dcreater · 2026-02-05 · in Yet another reminder why you should not use Ollama
  19. …terabyte of RAM and a single RTX6000. But to answer your question directly, tensor parallelism. https://github.com/ggml-org/llama.cpp/discussions/8735 https://docs.vllm.ai/en/latest/configuration/conserving_memo...
    comment by evilduck · 2026-01-27 · in Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model
  20. …that?) By using a grammar. Here is an example on how to only generate valid JSON with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... > A trillion is 8 symbols. You still haven't reached the end of your first…
    comment by johndough · 2026-01-20 · in I was a top 0.01% Cursor user, then switched to Claude Code 2.0
  21. …architecture and already supported by llama.cpp once the new name is added. Here's the PR: https://github.com/ggml-org/llama.cpp/pull/18936
    comment by cristoperb · 2026-01-19 · in GLM-4.7-Flash
  22. …for other backends in the README) using the Q4_K_M quant from ngxson on Lubuntu with an RTX 3090: https://github.com/ggml-org/llama.cpp/releases https://huggingface.co/ngxson/GLM-4.7-Flash-GGUF/blob/main/G...…
    comment by johndough · 2026-01-19 · in GLM-4.7-Flash
  23. it'a a new architecture. Not yet implemented in llama.cpp issue to follow: https://github.com/ggml-org/llama.cpp/issues/18931
    comment by homarp · 2026-01-19 · in GLM-4.7-Flash
  24. …llama.cpp supports GBNF grammars, which allows generation of JSON, or even programming languages: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
    comment by johndough · 2026-01-19 · in How to wrangle non-deterministic AI outputs into conventional software? (2025)
  25. …people know about the code and weights, since there are quite a few comments requesting support: https://github.com/ggml-org/llama.cpp/issues/16186
    comment by adefa · 2026-01-18 · in Flux 2 Klein pure C inference
  26. …less than a week. I submitted the work as a PR to the codebase and understandably, it was rejected. https://github.com/ggml-org/llama.cpp/pull/18404 https://huggingface.co/TrevorJS/Qwen3-Omni-30B-A3B-GGUF
    comment by adefa · 2026-01-18 · in Flux 2 Klein pure C inference
  27. …by default it’s automatic but has a much lower limit (I think 2/3 is total RAM or something). [0]: https://github.com/ggml-org/llama.cpp/discussions/2182#discu...
    comment by einsteinx2 · 2026-01-07 · in Intel Core Ultra Series 3 Debut as First Built on Intel 18A
  28. …in comparison, this isn't as rough as it sounds. Haven't had a chance to test that yet though. [1]: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... (using --mmproj AFAIK)
    comment by magicalhippo · 2026-01-02 · in Dell's version of the DGX Spark fixes pain points
  29. …can fit, such as gpt-oss-120, (gpt-oss-120b-mxfp4.gguf), which is about 60gb of weights (1). (1) https://github.com/ggml-org/llama.cpp/discussions/15396
    comment by kristianp · 2026-01-01 · in Dell's version of the DGX Spark fixes pain points
  30. you have a few 'claude' proxies on github llama.cpp recently added Anthropic API support https://github.com/ggml-org/llama.cpp/pull/17570
    comment by homarp · 2025-12-26 · in MiniMax M2.1: Built for Real-World Complex Tasks, Multi-Language Programming
  31. …optional Windows/Linux CUDA builds for faster inference. Everything runs locally. Big shout out to https://github.com/ggerganov/llama.cpp along with PyQt6, pdfplumber, pytesseract, and python-docx for making it easy to wire everything…
    comment by properbrew · 2025-12-23 · in Show HN: FormAIt – Free, private offline note formatting using LLMs
  32. …llama.cpp supports Anthropic's message format now[1], so you can use it with Claude Code[2]. [1]: https://github.com/ggml-org/llama.cpp/pull/17570 [2]: https://news.ycombinator.com/item?id=44654145
    comment by magicalhippo · 2025-12-19 · in LLM Year in Review
  33. …it on lambda.ai console or dashboard> Started building llama.cpp from source, history: 21 git clone https://github.com/ggml-org/llama.cpp 22 cd llama.cpp 23 which cmake 24 sudo apt list | grep libcurl 25 sudo apt-get install…
    comment by clusterhacks · 2025-12-10 · in Mistral releases Devstral2 and Mistral Vibe CLI
  34. …it's going to get better than the API integrations with google/bing that cloud models use. [0]: https://github.com/ggml-org/llama.cpp/discussions/4508
    comment by WatchDog · 2025-12-09 · in Apple's slow AI pace becomes a strength as market grows weary of spending
  35. …, https://github.com/huggingface/inference-playground/issues/1... and https://github.com/ggml-org/llama.cpp/discussions/15396#disc... .
    comment by mingodad · 2025-12-03 · in OpenAI declares 'code red' as Google catches up in AI race
  36. …noticeable slow but still very helpful). You can see the scrapped text for the whole seession here https://github.com/ggml-org/llama.cpp/discussions/17600
    comment by mingodad · 2025-11-29 · in So you wanna build a local RAG?
  37. …framework, not the model. E.g. https://docs.sglang.ai/advanced_features/deterministic_infer... https://github.com/ggml-org/llama.cpp/pull/16016
    comment by yorwba · 2025-11-15 · in Is it possible to have an LLM always give a consistent output?
  38. …JSON Schema and not something more flexible like BNF grammar's, like llama.cpp has for a long time: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
    comment by causal · 2025-11-14 · in Structured outputs on the Claude Developer Platform
  39. …barely use AI - just have a look at the development of support for Qwen3-Next-80B [1]. [1] https://github.com/ggml-org/llama.cpp/pull/16095
    comment by dust42 · 2025-11-08 · in Cerebras Code now supports GLM 4.6 at 1000 tokens/sec
  40. …results are very good and even weaker GPUs can run larger models at reasonable speeds. n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by IceWreck · 2025-11-02 · in Tongyi DeepResearch – open-source 30B MoE Model that rivals OpenAI DeepResearch
  41. …job as ML researcher,” but I still can’t justify $10k for “run a larger model on my Mac Studio” See https://github.com/ggml-org/llama.cpp/discussions/4167#discu...
    comment by gcr · 2025-11-01 · in My Impressions of the MacBook Pro M4
  42. With MoE models like gpt-oss, you can run some layers on the CPU (and some on GPU): https://github.com/ggml-org/llama.cpp/discussions/15396 Mentions 120b is runnable on 8GB VRAM too: "Note that even with just 8GB of VRAM, we can adjust the…
    comment by ThatPlayer · 2025-10-31 · in Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
  43. …get an idea of the speed difference, there is a benchmark page for llama.cpp on Apple silicon here: https://github.com/ggml-org/llama.cpp/discussions/4167 About quant selection: https://gist.github.com/Artefact2/b5f810600771265fc1e3944228..…
    comment by dust42 · 2025-10-31 · in Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
  44. …over this thing, and has the advantage of many years of already-working software. Apple benchmarks: https://github.com/ggml-org/llama.cpp/discussions/4167
    comment by suprjami · 2025-10-26 · in Nvidia DGX Spark: When benchmark numbers meet production reality
  45. …strange. There are official benchmarks of the Spark running multiple models just fine on llama.cpp https://github.com/ggml-org/llama.cpp/discussions/16578
    comment by veber-alex · 2025-10-26 · in Nvidia DGX Spark: When benchmark numbers meet production reality
  46. …Qwen3-next are also not enthused by LLM output. They had to go over everything and fix it up. https://github.com/ggml-org/llama.cpp/pull/16095#issuecommen...
    comment by samus · 2025-10-14 · in NanoChat – The best ChatGPT that $100 can buy
  47. Yes, I provided detailed numbers here: https://github.com/ggml-org/llama.cpp/discussions/16578
    comment by ggerganov · 2025-10-14 · in NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
  48. …is no. EDIT: Looks like Granite 4 hybrid architecture support was added to llama.cpp back in May: https://github.com/ggml-org/llama.cpp/pull/13550
    comment by RossBencina · 2025-10-03 · in 'Western Qwen': IBM Wows with Granite 4 LLM Launch and Hybrid Mamba/Transformer
  49. …on llama.cpp's grammars (nine hours and not a single mention of "GBNF"? HN is slipping) is here: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... There's also a grammar validation tool in the default llama.cpp build, which…
    comment by snerbles · 2025-09-23 · in Sampling and structured outputs in LLMs
  50. …active) on my 4080 and get a usable ~20 tk/s. Had to upgrade my system RAM, but that's easier. https://github.com/ggml-org/llama.cpp/discussions/15396 has a bit on getting that running
    comment by ThatPlayer · 2025-09-23 · in 25L Portable NV-linked Dual 3090 LLM Rig