ggml-org/llama.cpp on Hacker News

I googled it- supposed fixed template https://github.com/ggml-org/llama.cpp/issues/21347#issuecomm...

comment by jasonjmcghee · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

That also comes upstream from llama.cpp https://github.com/ggml-org/llama.cpp/discussions/4345

comment by wronglebowski · 2026-04-03 · in April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

…instead of JSON's double quotes, and disables json-to-schema conversion for these types." [1]: https://github.com/ggml-org/llama.cpp/pull/21326/commits/a50... edit: formatting

comment by airspresso · 2026-04-03 · in Google releases Gemma 4 open models

…an nVidia 16GB GPU, but Ollama's a bit easier to ease into since they have a preset model library. https://github.com/ggml-org/llama.cpp

comment by vunderba · 2026-04-02 · in Google releases Gemma 4 open models

…but even then barely enough for 64K for 24GB VRAM. Maybe 8 bit KV quantization is fine now after https://github.com/ggml-org/llama.cpp/pull/21038 got merged, so 100K+ is possible. > I should pick a full precision smaller model or 4 bit…

comment by petu · 2026-04-02 · in Google releases Gemma 4 open models

…with ROCm and Vulkan. These are sourced from https://github.com/lemonade-sdk/llamacpp-rocm and https://github.com/ggml-org/llama.cpp/releases respectively.

comment by sawansri · 2026-04-02 · in Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

…judge open models on launch day... it never goes well. There are ~always bugs on launch day. [0]: https://github.com/ggml-org/llama.cpp/pull/21326 [1]: https://github.com/ggml-org/llama.cpp/issues/21316

comment by coder543 · 2026-04-02 · in Google releases Gemma 4 open models

…Rotation. This seems to have genuinely helped KV cache compression as per [llama.cpp latest tests]( https://github.com/ggml-org/llama.cpp/pull/21038 ). On the other hand, [ik_llama.cpp]( https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/…

comment by 3abiton · 2026-04-02 · in Claude Code users hitting usage limits 'way faster than expected'

…(at least for prefill, which is a key bottleneck on pre-M5 platforms) is also in the works, per https://github.com/ggml-org/llama.cpp/issues/10453#issuecomm...

comment by zozbot234 · 2026-03-31 · in Ollama is now powered by MLX on Apple Silicon in preview

…than this. Starting with the llama.cpp discussion can give some better info than this blog post: https://github.com/ggml-org/llama.cpp/discussions/20969

comment by Aurornis · 2026-03-27 · in TurboQuant: Building a Sub-Byte KV Cache Quantizer from Paper to Production

▲ 4

Llama.cpp's Agents.md

story by Wowfunhappy · 2026-03-25 · 1 comment

▲ 6

Tell HN: Llamacpp now supports unified system RAM offloading on Linux

story by dabockster · 2026-03-24 · 0 comments

…really need to specify it as a CLI option.) OP has raised a discussion with the llama.cpp folks https://github.com/ggml-org/llama.cpp/discussions/20852 but little interest so far

comment by zozbot234 · 2026-03-24 · in Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

…being eminently public and applied across multiple relevant dimensions: raw inference speed ( https://github.com/ggml-org/llama.cpp/releases ), external-facing capabilities ( https://github.com/open-webui/open-webui/releases ) and…

comment by vntok · 2026-03-23 · in iPhone 17 Pro Demonstrated Running a 400B LLM

…days? I ran llama-bench a couple of weeks ago when there was a big speed improvement on llama.cpp ( https://github.com/ggml-org/llama.cpp/pull/20361#issuecommen... ): % llama-bench -m ~/ml-models/huggingface/ubergarm/Qwen3.5-397B-A17B-GGUF/…

comment by tarruda · 2026-03-22 · in Flash-MoE: Running a 397B Parameter Model on a Laptop

…extensive support for doing this and controlling exactly which weights end up where (llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/... , vllm: https://docs.vllm.ai/en/stable/configuration/engine_args/#of... ,…

comment by daemonologist · 2026-03-22 · in Flash-MoE: Running a 397B Parameter Model on a Laptop

Speculative decoding[1]? [1]: https://github.com/ggml-org/llama.cpp/blob/master/docs/specu...

comment by genpfault · 2026-03-14 · in Can I run AI locally?

https://github.com/ggml-org/llama.cpp/releases - has mac binaries https://unsloth.ai/docs/models/qwen3.5 - running locally guide for the Qwen 3.5 family of models, which have a range of different sizes.

comment by kristianp · 2026-03-14 · in Can I run AI locally?

…and checking it out. Fyi there's someone else on github keeping track of some LLM benchmarks too: https://github.com/ggml-org/llama.cpp/discussions/4167

comment by SkyMarshal · 2026-03-05 · in Show HN: Open dataset of real-world LLM performance on Apple Silicon

…in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...]…

comment by simonw · 2026-02-20 · in Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

This is the easiest set up on a Mac. You need at least 16gb on a MacBook: https://github.com/ggml-org/llama.cpp/discussions/15396

comment by ontouchstart · 2026-02-20 · in Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

▲ 839

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

story by lairv · 2026-02-20 · 223 comments

…It is a bug in the model weights and reproducible in their official chat UI. More details here: https://github.com/ggml-org/llama.cpp/pull/19283#issuecommen...

comment by tarruda · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed

…to generate the appropriate "I'm done thinking" indicator. You can see this described in this PR https://github.com/ggml-org/llama.cpp/pull/19635 Apparently Step 3.5 Flash uses an odd format for its tags so llama.cpp just doesn't handle it…

comment by sosodev · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed

…chains. - Current release has a bug where sometimes it goes into an infinite reasoning loop: https://github.com/ggml-org/llama.cpp/pull/19283#issuecommen... Hopefully StepFun will do a new release which addresses these issues. BTW StepFun…

comment by tarruda · 2026-02-19 · in Step 3.5 Flash – Open-source foundation model, supports deep reasoning at speed

Just in case anyone hasn't seen this yet: https://github.com/ggml-org/llama.cpp/discussions/15396 a guide for running gpt-oss on llama-server, with settings for various amounts of GPU memory, from 8GB on up

comment by Twirrim · 2026-02-15 · in IBM tripling entry-level jobs after finding the limits of AI adoption

…[1], which is useful if you are working with a custom programming language for example. [1] https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

comment by threeducks · 2026-02-05 · in Claude Opus 4.6

Georgi's relevant comment: https://github.com/ggml-org/llama.cpp/pull/19324#issuecommen...

comment by dcreater · 2026-02-05 · in Yet another reminder why you should not use Ollama

▲ 6

Yet another reminder why you should not use Ollama

story by dcreater · 2026-02-05 · 5 comments

▲ 1

Show HN: Notebook page on llama.cpp official webui

story by hleszek · 2026-02-04 · 0 comments

…terabyte of RAM and a single RTX6000. But to answer your question directly, tensor parallelism. https://github.com/ggml-org/llama.cpp/discussions/8735 https://docs.vllm.ai/en/latest/configuration/conserving_memo...

comment by evilduck · 2026-01-27 · in Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model

…that?) By using a grammar. Here is an example on how to only generate valid JSON with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R... > A trillion is 8 symbols. You still haven't reached the end of your first…

comment by johndough · 2026-01-20 · in I was a top 0.01% Cursor user, then switched to Claude Code 2.0

…architecture and already supported by llama.cpp once the new name is added. Here's the PR: https://github.com/ggml-org/llama.cpp/pull/18936

comment by cristoperb · 2026-01-19 · in GLM-4.7-Flash

…for other backends in the README) using the Q4_K_M quant from ngxson on Lubuntu with an RTX 3090: https://github.com/ggml-org/llama.cpp/releases https://huggingface.co/ngxson/GLM-4.7-Flash-GGUF/blob/main/G...…

comment by johndough · 2026-01-19 · in GLM-4.7-Flash

it'a a new architecture. Not yet implemented in llama.cpp issue to follow: https://github.com/ggml-org/llama.cpp/issues/18931

comment by homarp · 2026-01-19 · in GLM-4.7-Flash

…llama.cpp supports GBNF grammars, which allows generation of JSON, or even programming languages: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

comment by johndough · 2026-01-19 · in How to wrangle non-deterministic AI outputs into conventional software? (2025)

…people know about the code and weights, since there are quite a few comments requesting support: https://github.com/ggml-org/llama.cpp/issues/16186

comment by adefa · 2026-01-18 · in Flux 2 Klein pure C inference

…less than a week. I submitted the work as a PR to the codebase and understandably, it was rejected. https://github.com/ggml-org/llama.cpp/pull/18404 https://huggingface.co/TrevorJS/Qwen3-Omni-30B-A3B-GGUF

comment by adefa · 2026-01-18 · in Flux 2 Klein pure C inference

…by default it’s automatic but has a much lower limit (I think 2/3 is total RAM or something). [0]: https://github.com/ggml-org/llama.cpp/discussions/2182#discu...

comment by einsteinx2 · 2026-01-07 · in Intel Core Ultra Series 3 Debut as First Built on Intel 18A

…in comparison, this isn't as rough as it sounds. Haven't had a chance to test that yet though. [1]: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv... (using --mmproj AFAIK)

comment by magicalhippo · 2026-01-02 · in Dell's version of the DGX Spark fixes pain points

…can fit, such as gpt-oss-120, (gpt-oss-120b-mxfp4.gguf), which is about 60gb of weights (1). (1) https://github.com/ggml-org/llama.cpp/discussions/15396

comment by kristianp · 2026-01-01 · in Dell's version of the DGX Spark fixes pain points

you have a few 'claude' proxies on github llama.cpp recently added Anthropic API support https://github.com/ggml-org/llama.cpp/pull/17570

comment by homarp · 2025-12-26 · in MiniMax M2.1: Built for Real-World Complex Tasks, Multi-Language Programming

…optional Windows/Linux CUDA builds for faster inference. Everything runs locally. Big shout out to https://github.com/ggerganov/llama.cpp along with PyQt6, pdfplumber, pytesseract, and python-docx for making it easy to wire everything…

comment by properbrew · 2025-12-23 · in Show HN: FormAIt – Free, private offline note formatting using LLMs

…llama.cpp supports Anthropic's message format now[1], so you can use it with Claude Code[2]. [1]: https://github.com/ggml-org/llama.cpp/pull/17570 [2]: https://news.ycombinator.com/item?id=44654145

comment by magicalhippo · 2025-12-19 · in LLM Year in Review

…it on lambda.ai console or dashboard> Started building llama.cpp from source, history: 21 git clone https://github.com/ggml-org/llama.cpp 22 cd llama.cpp 23 which cmake 24 sudo apt list | grep libcurl 25 sudo apt-get install…

comment by clusterhacks · 2025-12-10 · in Mistral releases Devstral2 and Mistral Vibe CLI

…it's going to get better than the API integrations with google/bing that cloud models use. [0]: https://github.com/ggml-org/llama.cpp/discussions/4508

comment by WatchDog · 2025-12-09 · in Apple's slow AI pace becomes a strength as market grows weary of spending

…, https://github.com/huggingface/inference-playground/issues/1... and https://github.com/ggml-org/llama.cpp/discussions/15396#disc... .

comment by mingodad · 2025-12-03 · in OpenAI declares 'code red' as Google catches up in AI race

…noticeable slow but still very helpful). You can see the scrapped text for the whole seession here https://github.com/ggml-org/llama.cpp/discussions/17600

comment by mingodad · 2025-11-29 · in So you wanna build a local RAG?

…framework, not the model. E.g. https://docs.sglang.ai/advanced_features/deterministic_infer... https://github.com/ggml-org/llama.cpp/pull/16016

comment by yorwba · 2025-11-15 · in Is it possible to have an LLM always give a consistent output?

…JSON Schema and not something more flexible like BNF grammar's, like llama.cpp has for a long time: https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

comment by causal · 2025-11-14 · in Structured outputs on the Claude Developer Platform