Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …of its kind, OR it's the first good-enough one. I'm thinking I need to give this more testing! - https://github.com/ggml-org/llama.cpp/pull/31287 - https://github.com/ggml-org/llama.cpp/pull/31269
    comment by jakswa · 2026-10-28 · in Introducing LFM3: Every Modality for Specialized Intelligence
  2. …SQL statements when doing text-to-sql [2], and I think it's pretty cool. [0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars [1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c... [2] - https://gist.github.com
    comment by HanClinto · 2026-10-26 · in Fixing Tool-Call Accuracy in vLLM
  3. …on classified company materials. Here is an example problem what I daily report and deal with. https://github.com/ggml-org/llama.cpp/issues/27330
    comment by bozdemir · 2026-10-17 · in Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware
  4. …models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF " (or your inference engine and model of choice, walk away for a…
    comment by derefr · 2026-10-15 · in We’re releasing Jev 1.14.0 along with experimental features for dreamers
  5. …models is as easy as telling the agent you already use "set up laya on llama cpp locally https://github.com/ggml-org/llama.cpp https://huggingface.co/ggml-org/Laya-GGUF " (or your inference engine and model of choice, walk away for a…
    comment by Zambyte · 2026-10-15 · in We’re releasing Jev 1.14.0 along with experimental features for dreamers
  6. …the GGUF you launched it with. FWIW: llama-server's supported multiple models for a while now: https://github.com/ggml-org/llama.cpp/tree/master/tools/serv...
    comment by genpfault · 2026-10-09 · in Show HN: Apogee: Rebuilding Mozilla's Orbit, fully local and private
  7. …token, you can roughly infer for many models until you get better benchmarks. There is also: CUDA: https://github.com/ggml-org/llama.cpp/discussions/15013 HIP: https://github.com/ggml-org/llama.cpp/discussions/15021 Vulkan:…
    comment by DiabloD3 · 2026-10-08 · in How many GPUs is 1M/B/T tokens?
  8. They do make their standards explicit: https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTI... What part do you think is irrational gatekeeping?
    comment by anamexis · 2026-10-04 · in Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s
  9. The docs are a great start. https://github.com/ggml-org/llama.cpp/blob/master/docs/backe...
    comment by gunalx · 2026-10-02 · in Show HN: Janus – Go binary that runs GGUF models via Vulkan on AMD/Intel/Nvidia
  10. It looks like upstream work is underway: https://github.com/ggml-org/llama.cpp/issues/29058 The sheer number of geniuses working so fast on this repo never ceases to amaze me.
    comment by ranger_danger · 2026-09-24 · in Best LLM for every budget, updated daily
  11. …recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is…
    comment by utopiah · 2026-09-20 · in Qwen Image 2.1
  12. like https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ? At the risk of stating the obvious llama.cpp isn't just about LLaMa as https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.
    comment by utopiah · 2026-09-20 · in Qwen Image 2.1
  13. …output text, last time I checked (I do not know how to find a good source for that though). See https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the enum llm_arch { ...
    comment by mdp2021 · 2026-09-20 · in Qwen Image 2.1
  14. it's multimodal, see https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...
    comment by utopiah · 2026-09-20 · in Qwen Image 2.1
  15. …want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
    comment by francisjp · 2026-09-18 · in Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
  16. > I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 , works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices. p2p is disabled on nvidia customer…
    comment by my123 · 2026-09-14 · in CUDA for AMD on Windows
  17. …VK_NV_cooperative_matrix2 - extend? I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648 , works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
    comment by throwdbaaway · 2026-09-14 · in CUDA for AMD on Windows
  18. --no-mmproj-offload See the documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...
    comment by skirmish · 2026-09-09 · in Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses
  19. There's actually already a fork that implements the preliminaries: https://github.com/ggml-org/llama.cpp/discussions/21961
    comment by naasking · 2026-09-02 · in The efficient frontier of LLM inference
  20. …itself and whatever you're actually throwing at it. This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198
    comment by tharkun__ · 2026-09-01 · in Why your local LLM feels dumber than it is
  21. …gguf also fits and runs just fine on a cpu-only machine with 128GB RAM, using llama-server PR 27742 https://github.com/ggml-org/llama.cpp/pull/27742
    comment by walrus01 · 2026-08-27 · in Qwen3.8-Flash-Next
  22. Relevant PR: https://github.com/ggml-org/llama.cpp/pull/27742 This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8... cmake -B build -DGGML_CUDA=ON or cmake -B build -DGGML_METAL=ON then cmake --build build…
    comment by flounder3 · 2026-08-26 · in Qwen3.8-Flash-Next
  23. ▲ 3
    Gguf: Harden loader against malformed tensor dims and metadata types
    story by gzxharrison001 · 2026-08-26 · 0 comments
  24. …this GGUF's model architecture ('qwen4exp') Edit : Saw the pull request, should arrive soon enough https://github.com/ggml-org/llama.cpp/pull/27742
    comment by cwizou · 2026-08-26 · in Qwen3.8-Flash-Next
  25. …reading all the LLM generated comments and summaries from people was making me dizzy. RFC here https://github.com/ggml-org/llama.cpp/discussions/24528 which also links to some experimental implementations throughout the thread.
    comment by notnullorvoid · 2026-08-25 · in Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
  26. ▲ 4
    Release v0.2.0 · ggml-org/llama.cpp
    story by kyisaiah47 · 2026-08-25 · 0 comments
  27. …ld-Fusion-GAIN-V1.1-NM-DAU-NEO-MAX-MTP-GGUF incoai/Qwen3.8-27B-DFlash2-GGUF using llama.cpp PR/commit https://github.com/ggml-org/llama.cpp/pull/27342 on an AMD R9700 (32GB)
    comment by karmakaze · 2026-08-24 · in I Rented a 96 GB GPU and Took Uncensored Qwen3.8 From 44 to 125 tok/s
  28. …reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm... ) No inference engine is perfect, but I feel that llama.cpp is the most…
    comment by tarruda · 2026-08-23 · in Why your local LLM feels dumber than it is
  29. …tensors to maintain at higher accuracy, checking on representative data, and repeating. [0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...
    comment by kristjansson · 2026-08-19 · in Unsloth Dynamic 3.0 GGUFs
  30. llama.cpp PR https://github.com/ggml-org/llama.cpp/pull/27342
    comment by cogman10 · 2026-08-19 · in DFlash 2: Keep Drafting Parallel
  31. …line up with the actual version that's tracked inside the source code, which indicates v0.20.1: https://github.com/ggml-org/llama.cpp/commit/cea66f4c5a17255...
    comment by rpdillon · 2026-08-17
  32. …of this? There is also a v0.1.1 from 4 hours ago, but these v0.1.x one's are on the releases tab https://github.com/ggml-org/llama.cpp/releases seems like stray tags?
    comment by tingletech · 2026-08-17
  33. …... whether to use context shift on infinite text generation (default: disabled)" From: https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by pulse7 · 2026-08-14 · in Qwen 3.8 27B
  34. …^ --host 0.0.0.0 ^ --port 8033 [0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL) [1] https://github.com/ggml-org/llama.cpp/releases Instructions if you want to do the same: 1. download two files…
    comment by lelandbatey · 2026-08-14 · in Qwen 3.8 27B
  35. From https://github.com/ggml-org/llama.cpp : > Visit https://llama.app and follow the instructions It's linked at the start of the README.
    comment by gr_norm · 2026-08-12 · in llama.cpp
  36. …archives for just about any distribution and inference backend for the latest github release: https://github.com/ggml-org/llama.cpp/releases No need to compile unless you really need to.
    comment by epihelix · 2026-08-12 · in llama.cpp
  37. It's from https://github.com/ggml-org/llama.cpp -- not associated with Meta, it's been around for years, and surely they know about it -- so I would guess either it's not a trademark violation or they don't care.
    comment by reverius42 · 2026-08-12 · in llama.cpp
  38. …break things, rarely fix" (sorry, that's how it feels), bites here quite a bit. Two examples: - https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework…
    comment by imrehg · 2026-08-12 · in llama.cpp
  39. …of package repositories as a regular activity...). Git clone llama.cpp and build it, it's not hard. https://github.com/ggml-org/llama.cpp/blob/master/docs/build... literally just a few steps for the basics: git clone…
    comment by walrus01 · 2026-08-12 · in llama.cpp
  40. I was a bit suspicious of the url but it is also listed on llama.cpp github https://github.com/ggml-org/llama.cpp
    comment by tosh · 2026-08-12 · in llama.cpp
  41. I use llama.cpp w/ llama-swap https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap
    comment by computershit · 2026-08-10 · in Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
  42. …runs (cannot get MTP params working tho, about ~40 tok/s on my beefy 800GB/s 7900XT w/ 20GB VRAM). https://github.com/ggml-org/llama.cpp/pull/26841
    comment by jakswa · 2026-08-10 · in Muse Glimmer: 30B-parameter model optimized for always-on local agent workflows
  43. Seems it’s not fully supported in mainline llama.cpp yet https://github.com/ggml-org/llama.cpp/pull/25165 In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build…
    comment by andy99 · 2026-07-22 · in Laguna S 2.1
  44. Just merged into main llama.cpp a few hours ago: https://github.com/ggml-org/llama.cpp/pull/25165
    comment by suprjami · 2026-07-22 · in Laguna S 2.1
  45. …enable MTP (potentially 2x performance) on older GPUs (Kepler etc), so maybe it will work for you https://github.com/ggml-org/llama.cpp/pull/25680 Also, for Qwen, the 4 bit _XL quantization seems to have a good balance of performance to…
    comment by russianGuy83829 · 2026-07-14 · in Benchmarking 15 “E-Waste” GPUs with Modern Workloads
  46. …and it's not isolated to my setup. oMLX hit it, and llama.cpp has the same issue open right now ( https://github.com/ggml-org/llama.cpp/issues/22746 ). When two independent engines trip on the same thing, it usually points at a shared…
    comment by marzukia · 2026-07-12 · in Fixed three bugs that made Qwen3.5-122B a daily driver on Mac Studio
  47. …to see how HTTP became the primary transport, because it facilitated the motivating use-case. [1]: https://github.com/ggml-org/llama.cpp/pull/21674
    comment by magicalhippo · 2026-07-09 · in LLama.cpp Got Screwd
  48. …include a non-HTTP transport example, so seems clear to me it's not intended to be HTTP-only. [1]: https://github.com/ggml-org/llama.cpp/pull/24948
    comment by magicalhippo · 2026-07-09 · in LLama.cpp Got Screwd
  49. ▲ 11
    LLama.cpp Got Screwd
    story by trilogic · 2026-07-09 · 5 comments
  50. …sampler list: "(default: penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature)" [1] https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
    comment by pulse7 · 2026-07-03 · in Jamesob's guide to running SOTA LLMs locally