Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …thing they are doing. llama.cpp has supported this for a while ( using a BNF-style grammar -- https://github.com/ggerganov/llama.cpp/blob/master/grammars/... ) edit: oh actually, we do sort of know -- they call out jsonformer as an…
    comment by dilap · 2024-08-06 · in Structured Outputs in the API
  2. …has a lot of limitations, is not well documented, and can only be used indirectly through Core ML. https://github.com/ggerganov/llama.cpp/discussions/336
    comment by jcheng · 2024-07-24 · in Open source AI is the path forward
  3. …for whatever hardware they are supporting. For example here's the list of backends for Llama.cpp: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#su...
    comment by nl · 2024-07-23 · in Llama 3.1
  4. https://github.com/ollama/ollama/issues/5881 https://github.com/ggerganov/llama.cpp/issues/8650
    comment by jxy · 2024-07-23 · in Llama 3.1
  5. …know that ollama depends on llama.cpp. And as of now, llama.cpp doesn't support the new ROPE: https://github.com/ggerganov/llama.cpp/issues/8650 , and all ollama can do is wait for llama.cpp: https://github.com/ollama/ollama/issues/5881
    comment by jxy · 2024-07-23 · in Llama 3.1
  6. This 405B seriously need quantization solution like 1.625 bpw ternary packing for BitNet b1.58 https://github.com/ggerganov/llama.cpp/pull/8151
    comment by diimdeep · 2024-07-23 · in Llama 3.1
  7. …for this chips. Here is some grounded discussion I could find about AMD Ryzen AI NPU programming https://github.com/ggerganov/llama.cpp/issues/1499 From there, here is list of Ryzen AI engine enabled laptop/mini PC models…
    comment by diimdeep · 2024-07-21 · in AMD Ryzen AI 9 HX 370 12-Core APU Is Faster Than M3 Max
  8. Issue to track Mistral NeMo support in llama.cpp: https://github.com/ggerganov/llama.cpp/issues/8577
    comment by HanClinto · 2024-07-18 · in Mistral NeMo
  9. …to run models locally is to use Ollama with Continue.dev. This one is not available yet, though: https://github.com/ggerganov/llama.cpp/issues/8519
    comment by PufPufPuf · 2024-07-17 · in Codestral Mamba
  10. …meta.com/blog/large-language-model-llama-meta-ai/ [2] https://github.com/huggingface/transformers [3] https://github.com/ggerganov/llama.cpp [4] https://github.com/bitsandbytes-foundation/bitsandbytes [5]…
    comment by nabakin · 2024-07-16 · in Codestral Mamba
  11. …does not yet have Mamba2 support. The issues to track Mamba2 and Codestral Mamba support are here: https://github.com/ggerganov/llama.cpp/issues/8519 https://github.com/ggerganov/llama.cpp/issues/7727 Mamba support was added in March of…
    comment by HanClinto · 2024-07-16 · in Codestral Mamba
  12. …Face to their policy on “research and non-commercial use only” before downloading the model. https://github.com/ggerganov/llama.cpp/discussions/4576
    comment by wcallahan · 2024-07-12 · in Meta to release largest Llama 3 model on July 23 [405B]
  13. …Hacker News day after day. :) * [1] - https://github.com/HanClinto/mtg_belcher_montecarlo * [2] - https://github.com/ggerganov/llama.cpp/pull/6616
    comment by HanClinto · 2024-07-02 · in Getting the World Record in Hatetris (2022)
  14. ▲ 2
    Attention and final logit soft-capping, update scaling factor to Gemma2
    story by tosh · 2024-07-01 · 0 comments
  15. 4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9
    comment by helloericsf · 2024-06-20 · in Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC
  16. …to that question, but llama.cpp has a feature to constrain output to the provided grammar, such as https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Others should really implement that as well. You still need to guide the…
    comment by _flux · 2024-06-18 · in Every Way to Get Structured Output from LLMs
  17. …some great ideas and links. [1] https://platform.openai.com/docs/guides/text-generation/json... [2] https://github.com/ggerganov/llama.cpp/issues/4218
    comment by jmorgan · 2024-06-16 · in Ollama v0.1.45
  18. …llama.cpp is a one-liner. Check out the docker option if you don’t want to build/install llama.cpp. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by omneity · 2024-06-16 · in Ollama v0.1.45
  19. …glosses over explaining what feature they are talking about, here's a link to the docs about it: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by protosam · 2024-06-16 · in Ollama v0.1.45
  20. …case that shines. From this thread covering performance of llama.cpp on Apple Silicon M-series … https://github.com/ggerganov/llama.cpp/discussions/4167 … "Buy as much memory as you can afford would be my bottom line!"
    comment by Terretta · 2024-06-10 · in Apple Intelligence for iPhone, iPad, and Mac
  21. …text with even 7B models. I wonder if llama.cpp with SYCL could help, will have to try it out: https://github.com/ggerganov/llama.cpp/blob/master/README-sy... But even if that worked, I'd still have the problem that IDEs and whatever else…
    comment by KronisLV · 2024-06-04 · in Claude is unavailable for some users. Web and Mobile.
  22. …simply wouldn't be possible without the open models. [0] https://github.com/invoke-ai/InvokeAI [1] https://github.com/ggerganov/llama.cpp
    comment by lolinder · 2024-06-03 · in Why YC went to DC
  23. I think this comment explains it https://github.com/ggerganov/llama.cpp/discussions/4130#disc... As far as I understand (and mcharytoniuk should better confirm this), llama.cpp allows to chunk the context window of an LLM into independent…
    comment by kherud · 2024-06-02 · in Show HN: Open-source load balancer for llama.cpp
  24. …into smaller "slots", so it can process requests concurrently with continuous batching. See also: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by mcharytoniuk · 2024-06-02 · in Show HN: Open-source load balancer for llama.cpp
  25. and at some point ( https://github.com/ggerganov/llama.cpp/issues/7444 ) you will be able to use Phi-3-vision https://huggingface.co/microsoft/Phi-3-vision-128k-instruct but for now you will have to use python. You can try it here…
    comment by homarp · 2024-05-30 · in Ask HN: I have many PDFs – what is the best local way to leverage AI for search?
  26. …too repetitive. Some of them support getting very specific about the allowed output format, e.g. https://github.com/ggerganov/llama.cpp/blob/master/grammars/... So even if the LLM says that an invalid token is the most likely next token,…
    comment by sp332 · 2024-05-25 · in Mistral Fine-Tune
  27. ▲ 3
    Distributed LLM Inference with Llama.cpp
    story by tosh · 2024-05-24 · 0 comments
  28. …with all of that, you still won't really have a production-ready pipeline in the general case. [1] https://github.com/ggerganov/llama.cpp/issues/1300 [2] this is cutely called "constraining" a decoder; what it actually is is correcting a…
    comment by dvt · 2024-05-21 · in I want flexible queries, not RAG
  29. …game dev so I can comment on this but yes, packaging it locally would make the inference free. [1] https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by Isuckatcode · 2024-05-21 · in Ask HN: Which LLMs can run locally on most consumer computers
  30. …just processing - yes, and in fact 128 bits is far less than we're using already! If you look at https://github.com/ggerganov/llama.cpp you'll see this line: > AVX, AVX2 and AVX512 support for x86 architectures Guess what the 512 in AVX512…
    comment by yjftsjthsd-h · 2024-05-21 · in The prospects for 128 bit processors (John R. Mashey, 1995)
  31. ▲ 382
    New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy
    story by weinzierl · 2024-05-15 · 72 comments
  32. No, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by sp332 · 2024-05-15 · in Gemini Flash
  33. ▲ 2
    ggml: Add Flash Attention
    story by tosh · 2024-05-13 · 0 comments
  34. if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/... And you can search for…
    comment by homarp · 2024-05-08 · in IBM Granite: A Family of Open Foundation Models for Code Intelligence
  35. …as helix or llamacpp. helix: https://github.com/helix-editor/helix/blob/master/flake.nix llama: https://github.com/ggerganov/llama.cpp/blob/master/flake.nix > yet at the same time keeps all its packages in a single monolithic source tree…
    comment by rgoulter · 2024-05-07 · in Lix is a modern, delicious implementation of the Nix package manager
  36. …overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182
    comment by s1gsegv · 2024-05-06 · in Ask HN: Affordable hardware for running local large language models?
  37. …seen it being used in passing [0] - is this reasonable thing to do and we may see more of it? [0] https://github.com/ggerganov/llama.cpp/blob/master/build.zig
    comment by mirekrusin · 2024-05-05 · in Xmake: A cross-platform build utility based on Lua
  38. For anyone interested in exploring this, llama.cpp has an example implementation here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by HanClinto · 2024-05-01 · in Better and Faster Large Language Models via Multi-Token Prediction
  39. ▲ 2
    llama.cpp bfloat16 support
    story by indigodaddy · 2024-04-30 · 0 comments
  40. ▲ 3
    GGML Flash Attention support merged into llama.cpp
    story by smcleod · 2024-04-30 · 1 comment
  41. These numbers are a bit old but will give you a good ballpark for scaling: https://github.com/ggerganov/llama.cpp/discussions/4167 You can basically just divide by the multiple as you scale up parameters. Since this is all with a 7B model,…
    comment by lhl · 2024-04-26 · in Qwen1.5-110B
  42. …trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...
    comment by coder543 · 2024-04-24 · in Snowflake Arctic Instruct (128x3B MoE), largest open source model
  43. well https://github.com/ggerganov/llama.cpp/issues/6849 4k will work but "does not support long rope which is use by 128k variant."
    comment by homarp · 2024-04-23 · in Phi-3 Weights Released
  44. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by eshoyuan · 2024-04-22 · in Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
  45. …solution to drafting that does not require training, loading, and running draft LLMs. [1] https://github.com/ggerganov/llama.cpp/pull/2926
    comment by jncraton · 2024-04-21 · in Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
  46. ▲ 7
    Llama.cpp Working on Support for Llama3
    story by theolivenbaum · 2024-04-18 · 0 comments
  47. …Llama.cpp and the other models. When I was doing tests, I was comparing the Llama.cpp benchmarks ( https://github.com/ggerganov/llama.cpp/discussions/4167 ) to Apple's MPS - they match very closely. And then I was comparing Apple's MPS to…
    comment by kolinko · 2024-04-17 · in Show HN: Speeding up LLM inference 2x times (possibly)
  48. As always, code is the best documentation: https://github.com/ggerganov/llama.cpp/blob/8dd1ec8b3ffbfa2d...
    comment by adtac · 2024-04-17 · in Mixtral 8x22B
  49. Most inference servers are OpenAI-compatibile. Even the "official" llama-cpp server should work fine: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by talldayo · 2024-04-17 · in Mixtral 8x22B
  50. …ranked by number of commits. [0]: https://github.com/ollama/ollama/graphs/contributors and https://github.com/ggerganov/llama.cpp/graphs/contributors
    comment by bertman · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split