Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …support discussion, it's also worth noting that llama.cpp does not yet support gpu for mamba models https://github.com/ggerganov/llama.cpp/issues/6758
    comment by xyc · 2024-10-15 · in Zamba2-7B
  2. …Request: Support Zyphra/Zamba2-2.7B #8795 Open tomasmcm opened this issue on Jul 31 · 1 comment https://github.com/ggerganov/llama.cpp/issues/8795
    comment by kristianp · 2024-10-15 · in Zamba2-7B
  3. …to this, you should be able to leverage multi-GPU machines using the stock-and-standard Llama.cpp: https://github.com/ggerganov/llama.cpp/pull/1703
    comment by talldayo · 2024-10-12 · in Ask HN: AI at home with multiple graphics cards
  4. …that aside, SVE was shown to double that again, indicating there's significant headroom on NEON. ( https://github.com/ggerganov/llama.cpp/pull/9290 ) (I have ~0 idea what this is other than 'moar SIMD for ARM', for all I know, it's Amazon…
    comment by refulgentis · 2024-10-11 · in Initial CUDA Performance Lessons
  5. or with open source tools like llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/... or mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/... lmstudio and ollama use llama.cpp underneath. cut…
    comment by homarp · 2024-10-06 · in Show HN: Open-source real-time talk-to-AI wearable device for few $
  6. …issue that was opened on llama.cpp by a user who was concerned that the software had been "hacked": https://github.com/ggerganov/llama.cpp/issues/33#issuecommen... > Hey, I was reading your Readme.md and I saw that your repo was hacked. I…
    comment by HanClinto · 2024-10-04 · in Why does man print "gimme gimme gimme" at 00:30? (2017)
  7. https://github.com/ggerganov/llama.cpp/tree/master/examples/... you run that on the remote nodes
    comment by Palomides · 2024-10-03 · in Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
  8. …on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency ( https://github.com/ggerganov/llama.cpp/pull/7011 ). There's also a few tricks/updates I'd like to try which may improve performance, e.g.…
    comment by snep · 2024-10-03 · in AMD GPU Inference
  9. …/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp git checkout b3267 HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON…
    comment by slavik81 · 2024-10-02 · in AMD GPU Inference
  10. …ttps://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it…
    comment by qamononep · 2024-10-02 · in AMD GPU Inference
  11. You can use llama.cpp server's tokenize endpoint to tokenize and count the tokens: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by xyc · 2024-09-26 · in Llama 3.2: Revolutionizing edge AI and vision with open, customizable models
  12. Here is the feature request to support Pixtral on llama.cpp: https://github.com/ggerganov/llama.cpp/issues/9440 - once it's implemented in the upstream, ollama will be able to pick it up.
    comment by krasin · 2024-09-11 · in New Mistral AI Weights
  13. …sort of effect it might have to convince models that they really should stop and think first. [0]: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by HanClinto · 2024-09-06 · in Reflection 70B, the top open-source model
  14. llama.cpp enabled continuous batching by default half a year ago: https://github.com/ggerganov/llama.cpp/pull/6231 There is no need for a new API endpoint. Just send multiple requests at once.
    comment by johndough · 2024-09-03 · in Web scraping with GPT-4o: powerful but expensive
  15. …and compile llama.cpp and you should be able to run quantized models at an acceptable speed. https://github.com/ggerganov/llama.cpp Also, if you can spend the $60 and buy another 32GB of RAM, this will allow you to run the 30GB models…
    comment by programd · 2024-08-27 · in Cerebras launches inference for Llama 3.1; benchmarked at 1846 tokens/s on 8B
  16. …MiniCPM-V [5], etc. [1] - https://github.com/Dicklesworthstone/llm_aided_ocr/blob/main... [2] - https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#de... [3] - https://huggingface.co/microsoft/Phi-3-vision-128k-instruct [4] -…
    comment by daemonologist · 2024-08-09 · in Show HN: LLM-aided OCR – Correcting Tesseract OCR errors with LLMs
  17. …more egregious cases, but the memory load can be significant. One example of memory optimization: https://github.com/ggerganov/llama.cpp/pull/6616 I'm not entirely sure what other options there are for approaches to take, but I'd be…
    comment by HanClinto · 2024-08-06 · in Structured Outputs in the API
  18. Also note llama.cpp with grammar support: https://github.com/ggerganov/llama.cpp/tree/master/grammars Supports an EBNF-like syntax, as well as JSON-Schema.
    comment by HanClinto · 2024-08-06 · in Structured Outputs in the API
  19. …thing they are doing. llama.cpp has supported this for a while ( using a BNF-style grammar -- https://github.com/ggerganov/llama.cpp/blob/master/grammars/... ) edit: oh actually, we do sort of know -- they call out jsonformer as an…
    comment by dilap · 2024-08-06 · in Structured Outputs in the API
  20. …has a lot of limitations, is not well documented, and can only be used indirectly through Core ML. https://github.com/ggerganov/llama.cpp/discussions/336
    comment by jcheng · 2024-07-24 · in Open source AI is the path forward
  21. …for whatever hardware they are supporting. For example here's the list of backends for Llama.cpp: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su...
    comment by nl · 2024-07-23 · in Llama 3.1
  22. https://github.com/ollama/ollama/issues/5881 https://github.com/ggerganov/llama.cpp/issues/8650
    comment by jxy · 2024-07-23 · in Llama 3.1
  23. …know that ollama depends on llama.cpp. And as of now, llama.cpp doesn't support the new ROPE: https://github.com/ggerganov/llama.cpp/issues/8650 , and all ollama can do is wait for llama.cpp: https://github.com/ollama/ollama/issues/5881
    comment by jxy · 2024-07-23 · in Llama 3.1
  24. This 405B seriously need quantization solution like 1.625 bpw ternary packing for BitNet b1.58 https://github.com/ggerganov/llama.cpp/pull/8151
    comment by diimdeep · 2024-07-23 · in Llama 3.1
  25. …for this chips. Here is some grounded discussion I could find about AMD Ryzen AI NPU programming https://github.com/ggerganov/llama.cpp/issues/1499 From there, here is list of Ryzen AI engine enabled laptop/mini PC models…
    comment by diimdeep · 2024-07-21 · in AMD Ryzen AI 9 HX 370 12-Core APU Is Faster Than M3 Max
  26. Issue to track Mistral NeMo support in llama.cpp: https://github.com/ggerganov/llama.cpp/issues/8577
    comment by HanClinto · 2024-07-18 · in Mistral NeMo
  27. …to run models locally is to use Ollama with Continue.dev. This one is not available yet, though: https://github.com/ggerganov/llama.cpp/issues/8519
    comment by PufPufPuf · 2024-07-17 · in Codestral Mamba
  28. …meta.com/blog/large-language-model-llama-meta-ai/ [2] https://github.com/huggingface/transformers [3] https://github.com/ggerganov/llama.cpp [4] https://github.com/bitsandbytes-foundation/bitsandbytes [5]…
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  29. …does not yet have Mamba2 support. The issues to track Mamba2 and Codestral Mamba support are here: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/7727 Mamba support was added in March of…
    comment by HanClinto · 2024-07-16 · in Codestral Mamba
  30. …Face to their policy on “research and non-commercial use only” before downloading the model. https://github.com/ggerganov/llama.cpp/discussions/4576
    comment by wcallahan · 2024-07-12 · in Meta to release largest Llama 3 model on July 23 [405B]
  31. …Hacker News day after day. :) * [1] - https://github.com/HanClinto/mtg_belcher_montecarlo * [2] - https://github.com/ggerganov/llama.cpp/pull/6616
    comment by HanClinto · 2024-07-02 · in Getting the World Record in Hatetris (2022)
  32. 4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9
    comment by helloericsf · 2024-06-20 · in Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC
  33. …to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Others should really implement that as well. You still need to guide the…
    comment by _flux · 2024-06-18 · in Every Way to Get Structured Output from LLMs
  34. …some great ideas and links. [1] https://platform.openai.com/docs/guides/text-generation/json... [2] https://github.com/ggerganov/llama.cpp/issues/4218
    comment by jmorgan · 2024-06-16 · in Ollama v0.1.45
  35. …llama.cpp is a one-liner. Check out the docker option if you don’t want to build/install llama.cpp. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by omneity · 2024-06-16 · in Ollama v0.1.45
  36. …glosses over explaining what feature they are talking about, here's a link to the docs about it: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by protosam · 2024-06-16 · in Ollama v0.1.45
  37. …case that shines. From this thread covering performance of llama.cpp on Apple Silicon M-series … https://github.com/ggerganov/llama.cpp/discussions/4167 … "Buy as much memory as you can afford would be my bottom line!"
    comment by Terretta · 2024-06-10 · in Apple Intelligence for iPhone, iPad, and Mac
  38. …text with even 7B models. I wonder if llama.cpp with SYCL could help, will have to try it out: https://github.com/ggerganov/llama.cpp/blob/master/README-sy... But even if that worked, I'd still have the problem that IDEs and whatever else…
    comment by KronisLV · 2024-06-04 · in Claude is unavailable for some users. Web and Mobile.
  39. …simply wouldn't be possible without the open models. [0] https://github.com/invoke-ai/InvokeAI [1] https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2024-06-03 · in Why YC went to DC
  40. I think this comment explains it https://github.com/ggerganov/llama.cpp/discussions/4130#disc... As far as I understand (and mcharytoniuk should better confirm this), llama.cpp allows to chunk the context window of an LLM into independent…
    comment by kherud · 2024-06-02 · in Show HN: Open-source load balancer for llama.cpp
  41. …into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by mcharytoniuk · 2024-06-02 · in Show HN: Open-source load balancer for llama.cpp
  42. and at some point ( https://github.com/ggerganov/llama.cpp/issues/7444 ) you will be able to use Phi-3-vision https://huggingface.co/microsoft/Phi-3-vision-128k-instruct but for now you will have to use python. You can try it here…
    comment by homarp · 2024-05-30 · in Ask HN: I have many PDFs – what is the best local way to leverage AI for search?
  43. …too repetitive. Some of them support getting very specific about the allowed output format, e.g. https://github.com/ggerganov/llama.cpp/blob/master/grammars/... So even if the LLM says that an invalid token is the most likely next token,…
    comment by sp332 · 2024-05-25 · in Mistral Fine-Tune
  44. …with all of that, you still won't really have a production-ready pipeline in the general case. [1] https://github.com/ggerganov/llama.cpp/issues/1300 [2] this is cutely called "constraining" a decoder; what it actually is is correcting a…
    comment by dvt · 2024-05-21 · in I want flexible queries, not RAG
  45. …game dev so I can comment on this but yes, packaging it locally would make the inference free. [1] https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by Isuckatcode · 2024-05-21 · in Ask HN: Which LLMs can run locally on most consumer computers
  46. …just processing - yes, and in fact 128 bits is far less than we're using already! If you look at https://github.com/ggerganov/llama.cpp you'll see this line: > AVX, AVX2 and AVX512 support for x86 architectures Guess what the 512 in AVX512…
    comment by yjftsjthsd-h · 2024-05-21 · in The prospects for 128 bit processors (John R. Mashey, 1995)
  47. No, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by sp332 · 2024-05-15 · in Gemini Flash
  48. if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/... And you can search for…
    comment by homarp · 2024-05-08 · in IBM Granite: A Family of Open Foundation Models for Code Intelligence
  49. …as helix or llamacpp. helix: https://github.com/helix-editor/helix/blob/master/flake.nix llama: https://github.com/ggerganov/llama.cpp/blob/master/flake.nix > yet at the same time keeps all its packages in a single monolithic source tree…
    comment by rgoulter · 2024-05-07 · in Lix is a modern, delicious implementation of the Nix package manager
  50. …overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182
    comment by s1gsegv · 2024-05-06 · in Ask HN: Affordable hardware for running local large language models?