Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …barely use AI - just have a look at the development of support for Qwen3-Next-80B [1]. [1] https://github.com/ggml-org/llama.cpp/pull/16095
    comment by dust42 · 2025-11-08 · in Cerebras Code now supports GLM 4.6 at 1000 tokens/sec
  2. ▲ 4
    Llama.cpp launches official WebUI for local LLMs
    story by victormustar · 2025-11-04 · 0 comments
  3. …results are very good and even weaker GPUs can run larger models at reasonable speeds. n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by IceWreck · 2025-11-02 · in Tongyi DeepResearch – open-source 30B MoE Model that rivals OpenAI DeepResearch
  4. …job as ML researcher,” but I still can’t justify $10k for “run a larger model on my Mac Studio” See https://github.com/ggml-org/llama.cpp/discussions/4167#discu...
    comment by gcr · 2025-11-01 · in My Impressions of the MacBook Pro M4
  5. With MoE models like gpt-oss, you can run some layers on the CPU (and some on GPU): https://github.com/ggml-org/llama.cpp/discussions/15396 Mentions 120b is runnable on 8GB VRAM too: "Note that even with just 8GB of VRAM, we can adjust the…
    comment by ThatPlayer · 2025-10-31 · in Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
  6. …get an idea of the speed difference, there is a benchmark page for llama.cpp on Apple silicon here: https://github.com/ggml-org/llama.cpp/discussions/4167 About quant selection: https://gist.github.com/Artefact2/b5f810600771265fc1e3944228..…
    comment by dust42 · 2025-10-31 · in Ask HN: Who uses open LLMs and coding assistants locally? Share setup and laptop
  7. …over this thing, and has the advantage of many years of already-working software. Apple benchmarks: https://github.com/ggml-org/llama.cpp/discussions/4167
    comment by suprjami · 2025-10-26 · in Nvidia DGX Spark: When benchmark numbers meet production reality
  8. …strange. There are official benchmarks of the Spark running multiple models just fine on llama.cpp https://github.com/ggml-org/llama.cpp/discussions/16578
    comment by veber-alex · 2025-10-26 · in Nvidia DGX Spark: When benchmark numbers meet production reality
  9. …Qwen3-next are also not enthused by LLM output. They had to go over everything and fix it up. https://github.com/ggml-org/llama.cpp/pull/16095#issuecommen...
    comment by samus · 2025-10-14 · in NanoChat – The best ChatGPT that $100 can buy
  10. ▲ 2
    Guide: Setting up Nvidia DGX Spark with ggml
    story by homarp · 2025-10-14 · 0 comments
  11. Yes, I provided detailed numbers here: https://github.com/ggml-org/llama.cpp/discussions/16578
    comment by ggerganov · 2025-10-14 · in NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
  12. …is no. EDIT: Looks like Granite 4 hybrid architecture support was added to llama.cpp back in May: https://github.com/ggml-org/llama.cpp/pull/13550
    comment by RossBencina · 2025-10-03 · in 'Western Qwen': IBM Wows with Granite 4 LLM Launch and Hybrid Mamba/Transformer
  13. …on llama.cpp's grammars (nine hours and not a single mention of "GBNF"? HN is slipping) is here: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... There's also a grammar validation tool in the default llama.cpp build, which…
    comment by snerbles · 2025-09-23 · in Sampling and structured outputs in LLMs
  14. …active) on my 4080 and get a usable ~20 tk/s. Had to upgrade my system RAM, but that's easier. https://github.com/ggml-org/llama.cpp/discussions/15396 has a bit on getting that running
    comment by ThatPlayer · 2025-09-23 · in 25L Portable NV-linked Dual 3090 LLM Rig
  15. …limited you should go beyond 100 tokens/sec with the same model and a 8 bit quantization. 1. https://github.com/ggml-org/llama.cpp/discussions/15013
    comment by mysteria · 2025-09-18 · in Nvidia buys $5B in Intel
  16. ▲ 6
    Llama.cpp: Deterministic Inference Mode (CUDA): RMSNorm, MatMul, Attention
    story by diwank · 2025-09-15 · 0 comments
  17. From https://github.com/ggml-org/llama.cpp/issues/15940#issuecomm... : > This is a massive task, likely 2-3 months of full-time work for a highly specialized engineer. Until the Qwen team contributes the implementation, there are no quick…
    comment by NiekvdMaas · 2025-09-13 · in Qwen 3 now supports ARM and MLX
  18. https://github.com/ggml-org/llama.cpp/issues/15940
    comment by gre · 2025-09-12 · in Qwen3-Next
  19. This is the command probably: sudo sysctl iogpu.wired_limit_mb=184320 Source: https://github.com/ggml-org/llama.cpp/discussions/15396
    comment by frontsideair · 2025-09-09 · in Experimenting with Local LLMs on macOS
  20. …with both LPDDR5 and 32GB, only 16GB or LPDDR4(X). Using llama-bench, and Llama 2 7B Q4_0 like https://github.com/ggml-org/llama.cpp/discussions/10879 how does yours compare? Cuz I'm also comparing it with a few a few Ryzen 5 3000 Series…
    comment by ThatPlayer · 2025-09-08 · in Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5
  21. ▲ 2
    A better Llama-CLI help doc
    story by dcreater · 2025-09-02 · 1 comment
  22. …update apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y git clone https://github.com/ggerganov/llama.cpp cmake llama.cpp -B llama.cpp/build \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON cmake --build…
    comment by danielhanchen · 2025-08-24 · in DeepSeek-v3.1
  23. …grammar, the grammar constraints are applied to the full vocabulary and the token is resampled." https://github.com/ggml-org/llama.cpp/blob/54a241f505d515d62...
    comment by refulgentis · 2025-08-22 · in DeepSeek-v3.1
  24. ▲ 2
    Guide: Running GPT-OSS with Llama.cpp
    story by homarp · 2025-08-21 · 0 comments
  25. and in fact leverages that control to constrain outputs to those matching user-specified BNFs https://github.com/ggml-org/llama.cpp/tree/master/grammars
    comment by kristjansson · 2025-08-11 · in Ollama and gguf
  26. llama.cpp still doesn't support gpt-oss tool calling. https://github.com/ggml-org/llama.cpp/pull/15158 (among other similar PRs) But I also couldn't get vllm, or transformers serve, or ollama (400 response on /v1/chat/completions) working…
    comment by mhitza · 2025-08-11 · in Mistral Integration Improved in Llama.cpp
  27. It is fixed in this PR/branch: https://github.com/ggml-org/llama.cpp/pull/15181
    comment by tarruda · 2025-08-11 · in GPT-OSS-120B runs on just 8GB VRAM & 64GB+ system RAM
  28. ▲ 95
    Mistral Integration Improved in Llama.cpp
    story by decide1000 · 2025-08-11 · 15 comments
  29. …tokens for an AI model that's running on an Intel Arc, Nvidia 3090, and AMD gpu at the same time. https://github.com/ggml-org/llama.cpp/pull/5321
    comment by jychang · 2025-08-10 · in The Framework Desktop is a beast
  30. …Implementation for gpt-oss this week showed 2-3x improvements https://github.com/ggml-org/llama.cpp/pull/15157 https://www.reddit.com/r/LocalLLaMA/comments/1mkowrw/llamacp...
    comment by Maxious · 2025-08-09 · in How attention sinks keep language models stable
  31. And, as always, the FOSS ecosystem moves quickly, llama.cpp already fully support them! https://github.com/ggml-org/llama.cpp/pull/15157
    comment by diggan · 2025-08-08 · in How attention sinks keep language models stable
  32. Holy smokes, there's already llama.cpp support: https://github.com/ggml-org/llama.cpp/pull/15091
    comment by HanClinto · 2025-08-05 · in Open models by OpenAI
  33. ▲ 35
    Llama.cpp: Add GPT-OSS
    story by atgctg · 2025-08-05 · 0 comments
  34. …are we now? But not seeing any evidence that there's popular support in any of the main frameworks. https://github.com/ggml-org/llama.cpp/issues/1499 https://github.com/ollama/ollama/issues/5186 Good news, AMD has an initial implementation…
    comment by jauntywundrkind · 2025-07-30 · in My 2.5 year old laptop can write Space Invaders in JavaScript now (GLM-4.5 Air)
  35. ▲ 3
    AMD teams contributing to the llama.cpp codebase
    story by gzer0 · 2025-07-28 · 0 comments
  36. …incorrectly, since chatting with the model will use the actual correct token. Another is Llama 4 - https://github.com/ggml-org/llama.cpp/pull/12889 in which I fixed a RoPE issue - if we didn't fix it first, then again the calibration…
    comment by danielhanchen · 2025-07-23 · in Qwen3-Coder: Agentic coding in the world
  37. …they need to bump their llama.cpp runtime) The easiest would be to install llama.cpp from source: https://github.com/ggml-org/llama.cpp If you want to avoid it, I added SmolLM3 to MLX-LM as well: You can run it via `mlx_lm.chat --model…
    comment by reach-vb · 2025-07-09 · in Smollm3: Smol, multilingual, long-context reasoner LLM
  38. ▲ 2
    Hunyuan-A13B model support has been merged into llama.cpp
    story by empire23 · 2025-07-08 · 0 comments
  39. ▲ 5
    Is GitHub Releases Down?
    story by rvz · 2025-06-17 · 3 comments
  40. The average consumer uses llama.cpp. So here is your list of kernels: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/g... And here is pretty damning evidence that you're full of shit: https://github.com/ggml-org/llama.cpp/blob/m…
    comment by imtringued · 2025-06-16 · in AMD's AI Future Is Rack Scale 'Helios'
  41. Inference Library: An LLM inference library built over https://github.com/ggerganov/llama.cpp library for integrating LLMs into programs. Model Context Protocol Library: A C++ MCP client/server library that includes all fundamental…
    comment by xentnex · 2025-06-13 · in C++ MCP Client/Server SDK
  42. …applications. Here is a list of libraries: Inference Library: An LLM inference library built over https://github.com/ggerganov/llama.cpp library for integrating LLMs into programs. Model Context Protocol Library: An MCP client/server…
    comment by npalli · 2025-06-12 · in Mbase, an LLM SDK in C++
  43. > for near negligible drop in quality Hmm, that's evidently and anecdotally wrong: https://github.com/ggml-org/llama.cpp/discussions/4110
    comment by behnamoh · 2025-06-10 · in OpenAI dropped the price of o3 by 80%
  44. …power saving mode, inside ~8W. Come to think of it, those results might make a good blog post. [0] https://github.com/ggml-org/llama.cpp/discussions/10879 Search for "HX 370"
    comment by jakogut · 2025-06-06 · in AMD Radeon 8050S “Strix Halo” Linux Graphics Performance Review
  45. …of doing that, so I don't know how well the increased parameter size balances the quantization. See https://github.com/ggml-org/llama.cpp/pull/1684 and https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... for comparisons between…
    comment by rhdunn · 2025-05-30 · in Ask HN: What is the best LLM for consumer grade hardware?
  46. …started and probably all you need. I use it a lot on my M2 Macbook Air and it's really handy. [0] - https://github.com/ggml-org/llama.cpp [1] - https://lmstudio.ai/
    comment by thatcherc · 2025-05-21 · in Devstral
  47. …because this functionality has been in llama.cpp since 2023 if I understand this PR correctly: https://github.com/ggml-org/llama.cpp/pull/4312 Instead of offering a forked llama.cpp with the changes applied as commits, the repo wants you…
    comment by Aurornis · 2025-05-17 · in Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon
  48. > Citation? https://github.com/ollama/ollama/issues/7300#issuecomment-24... https://github.com/ggml-org/llama.cpp/blob/3e0be1cacef290c99... Anyway my point was just that it's not as easy as just pushing a patch upstream, like it is in many…
    comment by magicalhippo · 2025-05-17 · in Ollama's new engine for multimodal models
  49. “The llama.cpp project is the main playground for developing new features for the ggml library” -- https://github.com/ggml-org/llama.cpp “Some of the development is currently happening in the llama.cpp and whisper.cpp repos” --…
    comment by cwillu · 2025-05-16 · in Ollama's new engine for multimodal models
  50. FWIW, llama.cpp links to and fetches models from ollama ( https://github.com/ggml-org/llama.cpp/blob/master/tools/run/... ). This issue seems to be the typical case of someone being bothered for someone else, because it implies there's no…
    comment by levifig · 2025-05-16 · in Ollama violating llama.cpp license for over a year