Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …seen it being used in passing [0] - is this reasonable thing to do and we may see more of it? [0] https://github.com/ggerganov/llama.cpp/blob/master/build.zig
    comment by mirekrusin · 2024-05-05 · in Xmake: A cross-platform build utility based on Lua
  2. For anyone interested in exploring this, llama.cpp has an example implementation here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by HanClinto · 2024-05-01 · in Better and Faster Large Language Models via Multi-Token Prediction
  3. These numbers are a bit old but will give you a good ballpark for scaling: https://github.com/ggerganov/llama.cpp/discussions/4167 You can basically just divide by the multiple as you scale up parameters. Since this is all with a 7B model,…
    comment by lhl · 2024-04-26 · in Qwen1.5-110B
  4. …trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...
    comment by coder543 · 2024-04-24 · in Snowflake Arctic Instruct (128x3B MoE), largest open source model
  5. well https://github.com/ggerganov/llama.cpp/issues/6849 4k will work but "does not support long rope which is use by 128k variant."
    comment by homarp · 2024-04-23 · in Phi-3 Weights Released
  6. https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by eshoyuan · 2024-04-22 · in Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
  7. …solution to drafting that does not require training, loading, and running draft LLMs. [1] https://github.com/ggerganov/llama.cpp/pull/2926
    comment by jncraton · 2024-04-21 · in Lossless Acceleration of LLM via Adaptive N-Gram Parallel Decoding
  8. …Llama.cpp and the other models. When I was doing tests, I was comparing the Llama.cpp benchmarks ( https://github.com/ggerganov/llama.cpp/discussions/4167 ) to Apple's MPS - they match very closely. And then I was comparing Apple's MPS to…
    comment by kolinko · 2024-04-17 · in Show HN: Speeding up LLM inference 2x times (possibly)
  9. As always, code is the best documentation: https://github.com/ggerganov/llama.cpp/blob/8dd1ec8b3ffbfa2d...
    comment by adtac · 2024-04-17 · in Mixtral 8x22B
  10. Most inference servers are OpenAI-compatibile. Even the "official" llama-cpp server should work fine: https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by talldayo · 2024-04-17 · in Mixtral 8x22B
  11. …ranked by number of commits. [0]: https://github.com/ollama/ollama/graphs/contributors and https://github.com/ggerganov/llama.cpp/graphs/contributors
    comment by bertman · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
  12. …link to his profile and saw that he has opened some non-documentation pull requests for llama.cpp: https://github.com/ggerganov/llama.cpp/pull/5244 https://github.com/ggerganov/llama.cpp/pull/5576 I didn’t dig any deeper, but it took me…
    comment by JimDabell · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
  13. Quite possible that llama.cpp already supports WizardLM 2: https://github.com/ggerganov/llama.cpp/issues/6691
    comment by xyc · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
  14. llama.cpp can quantize a model for you: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#qu...
    comment by pja · 2024-04-11 · in Mistral AI Launches New 8x22B MOE Model
  15. …it towards the exact result). So this particular story doesn't need to be the same among all LLMs. https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by _flux · 2024-04-10 · in GPT-4 Turbo with Vision Generally Available
  16. …with ExLlama: https://github.com/turboderp/exllamav2/pull/275 And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...
    comment by cosmojg · 2024-04-08 · in Mixture-of-Depths: Dynamically allocating compute in transformers
  17. …where the base model is for completions and the instruct model for chat interactions. llama.cpp: https://github.com/ggerganov/llama.cpp/ deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...…
    comment by Metricon · 2024-04-07 · in The lifecycle of a code AI completion
  18. …dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of…
    comment by bt1a · 2024-04-07 · in More Agents Is All You Need: LLMs performance scales with the number of agents
  19. …would be visible with Github's new Activity tab, e.g., see the two force-pushes in llama.cpp https://github.com/ggerganov/llama.cpp/activity So, while, yes, git history can be rewritten, commits pushed to Github can effectively never be…
    comment by mxmlnkn · 2024-04-03 · in Reflections on Distrusting xz
  20. If anyone wants to follow along with EPYC optimizations, please subscribe to: https://github.com/ggerganov/llama.cpp/pull/6412#issuecommen...
    comment by jart · 2024-04-02 · in LLaMA now goes faster on CPUs
  21. it seems that BERT can be run on the llama.cpp platform https://github.com/ggerganov/llama.cpp/pull/5423 so presumably those models could benefit from the speed ups described in OP article when running on CPU
    comment by anentropic · 2024-04-02 · in LLaMA now goes faster on CPUs
  22. > https://github.com/ggerganov/llama.cpp/issues/91 > Justine was kicked out of llama.cpp for introducing changes before the rest of the maintainers approved them. I followed your link and found nothing regarding this. What am I missing?…
    comment by cryptonector · 2024-04-01 · in LLaMA now goes faster on CPUs
  23. Plenty of claims about it, e.g. here as a "fact": https://github.com/ggerganov/llama.cpp/discussions/638#discu... . I don't think occasional expressions of lingering doubt (still couched among positive language like calling it a "miracle")…
    comment by 4bpp · 2024-04-01 · in LLaMA now goes faster on CPUs
  24. Here are links to the most recent pull requests sent https://github.com/ggerganov/llama.cpp/pull/6414 https://github.com/ggerganov/llama.cpp/pull/6412
    comment by homarp · 2024-04-01 · in LLaMA now goes faster on CPUs
  25. …capabilities, making even old parts competitive. There are two PRs raised to merge to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/6414 https://github.com/ggerganov/llama.cpp/pull/6412 Hopefully these can be accepted, without…
    comment by dagaci · 2024-04-01 · in LLaMA now goes faster on CPUs
  26. The PR associated with the issue was merged after the approval of the owner of the repo ( https://github.com/ggerganov/llama.cpp/pull/613 ). And there is other more recent contributions. Where do you see any drama ?
    comment by Rexxar · 2024-04-01 · in LLaMA now goes faster on CPUs
  27. …the public cloud, developers are still targeting AVX512 when they should be targeting VNNI and AMX. https://github.com/ggerganov/llama.cpp/issues/2555
    comment by kiratp · 2024-04-01 · in LLaMA now goes faster on CPUs
  28. https://github.com/ggerganov/llama.cpp/issues/91 Justine was kicked out of llama.cpp for introducing changes before the rest of the maintainers approved them. Much drama has been had since then and I've lost interest in both projects. It's…
    comment by llm_trw · 2024-04-01 · in LLaMA now goes faster on CPUs
  29. llama.cpp probably won't be getting Jamba support anytime soon: https://github.com/ggerganov/llama.cpp/issues/6372#issuecomm... There is an MLX Mamba implementation, but nothing for Jamba either: https://github.com/alxndrTL/mamba.py/tree/ma…
    comment by lhl · 2024-03-30 · in Jamba: Production-grade Mamba-based AI model
  30. git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./server -m models/7B/ggml-model.gguf -c 2048 I don't think it'll take you the whole weekend :)
    comment by gryfft · 2024-03-27 · in Claude 3 surpasses GPT-4 on Chatbot Arena for the first time
  31. …llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and…
    comment by wokwokwok · 2024-03-27 · in DBRX: A new open LLM
  32. …via 'Nitro', which was discussed here before [2]. [0] https://github.com/vllm-project/vllm [1] https://github.com/ggerganov/llama.cpp [2] https://news.ycombinator.com/item?id=38887531
    comment by lagrange77 · 2024-03-21 · in Jan: An open source alternative to ChatGPT that runs on the desktop
  33. Direct link to releases: https://github.com/ggerganov/llama.cpp/releases
    comment by schappim · 2024-03-21 · in Llama.cpp releases now ship with pre-built macOS binaries
  34. …which run on CPU but can have some layers offloaded on GPU. I definitely recommend reading the https://github.com/ggerganov/llama.cpp documentation.
    comment by fbhabbed · 2024-03-18 · in Ask HN: How to get started with local language models?
  35. If you're just looking to test it out, it's probably easiest to wait for llama.cpp to add support ( https://github.com/ggerganov/llama.cpp/issues/6120 ), and then you can run it slowly if you have enough RAM, or wait for one of the…
    comment by zone411 · 2024-03-18 · in Grok
  36. …a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x…
    comment by jameshart · 2024-03-15 · in Ollama now supports AMD graphics cards
  37. Here is the commit that added ROCm support to llama.cpp back in August: https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
    comment by rahimnathwani · 2024-03-15 · in Ollama now supports AMD graphics cards
  38. …Vulkan thread for instance I see people getting it working with Polaris and even Hawaii cards. https://github.com/ggerganov/llama.cpp/pull/2059 Personally I just run it on CPU and several tokens/s is good enough for my purposes.
    comment by mysteria · 2024-03-15 · in Ollama now supports AMD graphics cards
  39. llama.cpp also supports AMD cards: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi...
    comment by Zambyte · 2024-03-15 · in Ollama now supports AMD graphics cards
  40. I'm using https://github.com/ggerganov/llama.cpp and currently mistral 7b (on a m1 macbook pro). I'm sure with some prompt examples you can get pretty good results on a smaller model. At the moment I don't have it open sourced due to it…
    comment by cheesehead · 2024-03-07 · in Show HN: Tech jobs on the command line
  41. …in GPUs. Does anyone have any idea what the bit patterns are for (starting at line 1529)? [1] https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda...
    comment by qrios · 2024-03-01 · in GGUF, the Long Way Around
  42. …it's still worth it all the way down to 2-bit. The chart in this llama.cpp PR speaks for itself: https://github.com/ggerganov/llama.cpp/pull/1684#issue-17396...
    comment by int_19h · 2024-02-29 · in The Era of 1-bit LLMs: ternary parameters for cost-effective computing
  43. These may be useful https://github.com/ggerganov/llama.cpp/blob/master/grammars https://github.com/guidance-ai/guidance?tab=readme-ov-file#c... https://github.com/eth-sri/lmql/issues/172
    comment by isaacfung · 2024-02-27 · in Show HN: Supermaven, the first code completion tool with 300k token context
  44. …just got merged, so it may take some time (could be hours or days) until this lands in ollama https://github.com/ggerganov/llama.cpp/pull/5631
    comment by SushiHippie · 2024-02-21 · in Gemma: New Open Models
  45. It should be possible to run it via llama.cpp[0] now. [0] https://github.com/ggerganov/llama.cpp/pull/5631
    comment by Kostic · 2024-02-21 · in Gemma: New Open Models
  46. If you just check out https://github.com/ggerganov/llama.cpp and run make, you’ll wind up with an executable called ‘main’ that lets you run any gguf language model you choose. Then: ./main -m ./models/30B/llama-30b.Q4_K_M.gguf --prompt…
    comment by jameshart · 2024-02-18 · in Ollama is now available on Windows in preview
  47. …will load just fine into ollama: https://old.reddit.com/r/LocalLLaMA/comments/18av9aw/quick_s... https://github.com/ggerganov/llama.cpp/discussions/2948 You can run ollama (and a web UI) pretty trivially via docker: docker run -d…
    comment by okwhateverdude · 2024-02-14 · in World model on million-length video and language with RingAttention
  48. …CUDA-compatibility layer - ZLUDA on Radeon (and not yet Intel OneAPI) - instead or in addition? https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi... https://news.ycombinator.com/item?id=38588573 What can't WebGPU abstract away…
    comment by westurner · 2024-02-12 · in AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source
  49. …- llama benchmarked on all M-series chips, and in comments there are comparisons with nvidia. https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by kolinko · 2024-02-11 · in Ask HN: Are there any reliable benchmarks for Machine Learning Model Serving?
  50. Is there a tutorial on how to get that setup running step-by-step? I only found a GitHub issue ( https://github.com/ggerganov/llama.cpp/issues/4439 ) that mentions that mainline llama.cpp isn't working for the model. Bonus question if you…
    comment by codeisawesome · 2024-02-09 · in Goody-2, the world's most responsible AI model