Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. https://github.com/ggerganov/llama.cpp/discussions/9565 Discussions under https://news.ycombinator.com/item?id=40693391 (I recommend doing a search yourself first) Basically, if you know how to use a computer, you can use Ollama (almost).…
    comment by n144q · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  2. Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
    comment by Kubuxu · 2025-01-31 · in Llama.cpp supports Vulkan. why doesn't Ollama?
  3. The answer to your question is yes. There is an open issue with llama.cpp about this very thing: https://github.com/ggerganov/llama.cpp/issues/11333 The TLDR is that llama.cpp’s NUMA support is suboptimal, which is hurting performance…
    comment by ryao · 2025-01-29 · in Complete hardware and software setup for running Deepseek-R1 locally
  4. …Additionally, llama.cpp’s NUMA support is suboptimal, so he is likely taking a performance hit: https://github.com/ggerganov/llama.cpp/issues/11333 When llama.cpp fixes its NUMA support, using two sockets should be no worse than using one…
    comment by ryao · 2025-01-29 · in Complete hardware and software setup for running Deepseek-R1 locally
  5. …replaced by the model. The heck, even your SIMD expert now seems to be replaced by the model too ( https://github.com/ggerganov/llama.cpp/pull/11453/files ). Those companies will no longer need M + N + ... engineers to create the business…
    comment by menaerus · 2025-01-29 · in Promising results from DeepSeek R1 for code
  6. ▲ 1
    Ggml: X2 speed for WASM by optimizing SIMD
    story by btilly · 2025-01-28 · 2 comments
  7. …of a typical change request by Java Developer #1753 at Fortune 500 Enterprise Company #271. [0] https://github.com/ggerganov/llama.cpp/pull/11453/files
    comment by mjr00 · 2025-01-28 · in Promising results from DeepSeek R1 for code
  8. ▲ 6
    DeepSeek-R1 speeds up llama.cpp code by x2
    story by roboboffin · 2025-01-28 · 3 comments
  9. ▲ 4
    Llama.cpp PR with 99% of code written by DeepSeek-R1
    story by zelag · 2025-01-28 · 0 comments
  10. > vs most things related to hardware or low level work. counter point: https://github.com/ggerganov/llama.cpp/pull/11453 > This PR provides a big jump in speed for WASM by leveraging SIMD instructions for qX_K_q8_K and qX_0_q8_0 dot…
    comment by NitpickLawyer · 2025-01-27 · in I trusted an LLM, now I'm on day 4 of an afternoon project
  11. ▲ 7
    Ggml 2x WASM Speed with SIMD Optimization Using 99% DeekSeek-R1-Generated Code
    story by bratao · 2025-01-27 · 0 comments
  12. …nice idea. Does anyone know if anyone else does something similar? [0] https://www.tabbyml.com/ [1] https://github.com/ggerganov/llama.cpp/pull/9787#issue-25729... "global context onwards" [2] https://tabby.tabbyml.com/docs/administration/c…
    comment by mijoharas · 2025-01-24 · in Llama.vim – Local LLM-assisted text completion
  13. …has improved compared to what is seen in the README videos thanks to client-side caching. [0] - https://github.com/ggerganov/llama.cpp/pull/9787 [1] - https://github.com/ggml-org/llama.vscode
    comment by ggerganov · 2025-01-23 · in Llama.vim – Local LLM-assisted text completion
  14. …image processing requirements AFIAK. [1]: https://github.com/ollama/ollama/releases/tag/v0.4.0 [2]: https://github.com/ggerganov/llama.cpp/issues/9643
    comment by magicalhippo · 2025-01-12 · in Phi 4 available on Ollama
  15. …output to tokens with the desired prefix, probably with some mechanism similar to how this works: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by do_not_redeem · 2025-01-11 · in Character Prefix Conditioning
  16. …runs reasonably well on a 24GB GPU (7900XTX or 4090) and 64GB of regular RAM. Software: https://github.com/ggerganov/llama.cpp .
    comment by skirmish · 2024-12-26 · in GPT-5 is behind schedule
  17. Compare performance on various Macs here as it gets updated: https://github.com/ggerganov/llama.cpp/discussions/4167 OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling…
    comment by Terretta · 2024-12-15 · in Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning
  18. ▲ 155
    Llama.cpp Now Supports Qwen2-VL (Vision Language Model)
    story by BUFU · 2024-12-14 · 50 comments
  19. …offers significant speed improvements, which I understand can sometimes be shortcoming of GBNF (e.g https://github.com/ggerganov/llama.cpp/issues/4218 ).
    comment by mwieler · 2024-12-08 · in Structured Outputs with Ollama
  20. …[1] https://github.com/guidance-ai/llguidance/blob/main/parser/s... [2] https://github.com/ggerganov/llama.cpp/pull/10224 [3] https://github.com/guidance-ai/llgtrt
    comment by mmoskal · 2024-12-07 · in Structured Outputs with Ollama
  21. If anyone needs a more powerful constrain outputs, llama.cpp support gbnf: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by rdescartes · 2024-12-07 · in Structured Outputs with Ollama
  22. …line in the end? [1] - https://github.com/ollama/ollama/commit/1bdab9fdb19f8a8c73ed... [2] - since https://github.com/ggerganov/llama.cpp/commit/bcc0eb4591bec5...
    comment by wokwokwok · 2024-12-05 · in Bringing K/V context quantisation to Ollama
  23. …a good way to get confidence scores in llama.cpp (Been tracking this git request to get log_probs https://github.com/ggerganov/llama.cpp/issues/6423 )
    comment by inLewOf · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  24. …l-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true # build llama.cpp git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang++-17 cmake -S. -Bbuild \ -DGGML_HIPBLAS=ON \…
    comment by slavik81 · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  25. You can get a release binary from https://github.com/ggerganov/llama.cpp/releases too.
    comment by xyc · 2024-11-30 · in Llama.cpp guide – Running LLMs locally on any hardware, from scratch
  26. …from the fp16 variant. [1] https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF [2] https://github.com/ggerganov/llama.cpp
    comment by johndough · 2024-11-10 · in OpenCoder: Open Cookbook for Top-Tier Code Large Language Models
  27. No need to use Ollama. LLama.cpp has its own OpenAI-compatible server[0] and it works great. [0] https://github.com/ggerganov/llama.cpp#web-server
    comment by milansuk · 2024-10-30 · in Go library for in-process vector search and embeddings with llama.cpp
  28. …similar experience - but did find I got better output by forcing them to adhere to a fixed grammar: https://github.com/ggerganov/llama.cpp/tree/master/grammars For context, I was playing with a script to bulk download podcasts, transcribe…
    comment by wswope · 2024-10-24 · in Quantized Llama models with increased speed and a reduced memory footprint
  29. ▲ 2
    Llama.vim: Plugin for Neovim
    story by mariuz · 2024-10-22 · 0 comments
  30. ▲ 2
    Llama.vim: Plugin for Neovim
    story by ibobev · 2024-10-21 · 0 comments
  31. …There are a variety of other LLM inference implementations that can run on CPU as well. [0] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su... [1] - https://docs.vllm.ai/en/v0.6.1/getting_started/cpu-installat...
    comment by kkielhofner · 2024-10-18 · in Ask HN: I have 24 core server with 1TB of DDR4 RAM, what should I run?
  32. …non-deterministic (maybe stochastic) output, but maybe we will see it in the next few years. [1] https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by Lockal · 2024-10-18 · in The Prompt() Function: Use the Power of LLMs with SQL
  33. …support discussion, it's also worth noting that llama.cpp does not yet support gpu for mamba models https://github.com/ggerganov/llama.cpp/issues/6758
    comment by xyc · 2024-10-15 · in Zamba2-7B
  34. …Request: Support Zyphra/Zamba2-2.7B #8795 Open tomasmcm opened this issue on Jul 31 · 1 comment https://github.com/ggerganov/llama.cpp/issues/8795
    comment by kristianp · 2024-10-15 · in Zamba2-7B
  35. …to this, you should be able to leverage multi-GPU machines using the stock-and-standard Llama.cpp: https://github.com/ggerganov/llama.cpp/pull/1703
    comment by talldayo · 2024-10-12 · in Ask HN: AI at home with multiple graphics cards
  36. …that aside, SVE was shown to double that again, indicating there's significant headroom on NEON. ( https://github.com/ggerganov/llama.cpp/pull/9290 ) (I have ~0 idea what this is other than 'moar SIMD for ARM', for all I know, it's Amazon…
    comment by refulgentis · 2024-10-11 · in Initial CUDA Performance Lessons
  37. or with open source tools like llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/... or mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/... lmstudio and ollama use llama.cpp underneath. cut…
    comment by homarp · 2024-10-06 · in Show HN: Open-source real-time talk-to-AI wearable device for few $
  38. …issue that was opened on llama.cpp by a user who was concerned that the software had been "hacked": https://github.com/ggerganov/llama.cpp/issues/33#issuecommen... > Hey, I was reading your Readme.md and I saw that your repo was hacked. I…
    comment by HanClinto · 2024-10-04 · in Why does man print "gimme gimme gimme" at 00:30? (2017)
  39. https://github.com/ggerganov/llama.cpp/tree/master/examples/... you run that on the remote nodes
    comment by Palomides · 2024-10-03 · in Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
  40. …on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency ( https://github.com/ggerganov/llama.cpp/pull/7011 ). There's also a few tricks/updates I'd like to try which may improve performance, e.g.…
    comment by snep · 2024-10-03 · in AMD GPU Inference
  41. …/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON…
    comment by slavik81 · 2024-10-02 · in AMD GPU Inference
  42. …ttps://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it…
    comment by qamononep · 2024-10-02 · in AMD GPU Inference
  43. You can use llama.cpp server's tokenize endpoint to tokenize and count the tokens: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by xyc · 2024-09-26 · in Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
  44. Here is the feature request to support Pixtral on llama.cpp: https://github.com/ggerganov/llama.cpp/issues/9440 - once it's implemented in the upstream, ollama will be able to pick it up.
    comment by krasin · 2024-09-11 · in New Mistral AI Weights
  45. …sort of effect it might have to convince models that they really should stop and think first. [0]: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by HanClinto · 2024-09-06 · in Reflection 70B, the top open-source model
  46. llama.cpp enabled continuous batching by default half a year ago: https://github.com/ggerganov/llama.cpp/pull/6231 There is no need for a new API endpoint. Just send multiple requests at once.
    comment by johndough · 2024-09-03 · in Web scraping with GPT-4o: powerful but expensive
  47. …and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models…
    comment by programd · 2024-08-27 · in Cerebras launches inference for Llama 3.1; benchmarked at 1846 tokens/s on 8B
  48. …MiniCPM-V [5], etc. [1] - https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main... [2] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#de... [3] - https://huggingface.co/microsoft/Phi-3-vision-128k-instruct [4] -…
    comment by daemonologist · 2024-08-09 · in Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
  49. …more egregious cases, but the memory load can be significant. One example of memory optimization: https://github.com/ggerganov/llama.cpp/pull/6616 I'm not entirely sure what other options there are for approaches to take, but I'd be…
    comment by HanClinto · 2024-08-06 · in Structured Outputs in the API
  50. Also note llama.cpp with grammar support: https://github.com/ggerganov/llama.cpp/tree/master/grammars Supports an EBNF-like syntax, as well as JSON-Schema.
    comment by HanClinto · 2024-08-06 · in Structured Outputs in the API