Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 1
    Llama.cpp: Improve CPU prompt eval speed
    story by tosh · 2024-04-17 · 0 comments
  2. …link to his profile and saw that he has opened some non-documentation pull requests for llama.cpp: https://github.com/ggerganov/llama.cpp/pull/5244 https://github.com/ggerganov/llama.cpp/pull/5576 I didn’t dig any deeper, but it took me…
    comment by JimDabell · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
  3. Quite possible that llama.cpp already supports WizardLM 2: https://github.com/ggerganov/llama.cpp/issues/6691
    comment by xyc · 2024-04-17 · in Ollama 0.1.32: WizardLM 2, Mixtral 8x22B, macOS CPU/GPU model split
  4. llama.cpp can quantize a model for you: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#qu...
    comment by pja · 2024-04-11 · in Mistral AI Launches New 8x22B MOE Model
  5. …it towards the exact result). So this particular story doesn't need to be the same among all LLMs. https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by _flux · 2024-04-10 · in GPT-4 Turbo with Vision Generally Available
  6. …with ExLlama: https://github.com/turboderp/exllamav2/pull/275 And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...
    comment by cosmojg · 2024-04-08 · in Mixture-of-Depths: Dynamically allocating compute in transformers
  7. …where the base model is for completions and the instruct model for chat interactions. llama.cpp: https://github.com/ggerganov/llama.cpp/ deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...…
    comment by Metricon · 2024-04-07 · in The lifecycle of a code AI completion
  8. …dynamic) component One example from the llama.cpp codebase is dynamic temperature sampling https://github.com/ggerganov/llama.cpp/pull/4972/files Not sure what you mean by whole model state given that there are tens of thousands of…
    comment by bt1a · 2024-04-07 · in More Agents Is All You Need: LLMs performance scales with the number of agents
  9. …would be visible with Github's new Activity tab, e.g., see the two force-pushes in llama.cpp https://github.com/ggerganov/llama.cpp/activity So, while, yes, git history can be rewritten, commits pushed to Github can effectively never be…
    comment by mxmlnkn · 2024-04-03 · in Reflections on Distrusting xz
  10. If anyone wants to follow along with EPYC optimizations, please subscribe to: https://github.com/ggerganov/llama.cpp/pull/6412#issuecommen...
    comment by jart · 2024-04-02 · in LLaMA now goes faster on CPUs
  11. it seems that BERT can be run on the llama.cpp platform https://github.com/ggerganov/llama.cpp/pull/5423 so presumably those models could benefit from the speed ups described in OP article when running on CPU
    comment by anentropic · 2024-04-02 · in LLaMA now goes faster on CPUs
  12. > https://github.com/ggerganov/llama.cpp/issues/91 > Justine was kicked out of llama.cpp for introducing changes before the rest of the maintainers approved them. I followed your link and found nothing regarding this. What am I missing?…
    comment by cryptonector · 2024-04-01 · in LLaMA now goes faster on CPUs
  13. Plenty of claims about it, e.g. here as a "fact": https://github.com/ggerganov/llama.cpp/discussions/638#discu... . I don't think occasional expressions of lingering doubt (still couched among positive language like calling it a "miracle")…
    comment by 4bpp · 2024-04-01 · in LLaMA now goes faster on CPUs
  14. Here are links to the most recent pull requests sent https://github.com/ggerganov/llama.cpp/pull/6414 https://github.com/ggerganov/llama.cpp/pull/6412
    comment by homarp · 2024-04-01 · in LLaMA now goes faster on CPUs
  15. …capabilities, making even old parts competitive. There are two PRs raised to merge to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/6414 https://github.com/ggerganov/llama.cpp/pull/6412 Hopefully these can be accepted, without…
    comment by dagaci · 2024-04-01 · in LLaMA now goes faster on CPUs
  16. The PR associated with the issue was merged after the approval of the owner of the repo ( https://github.com/ggerganov/llama.cpp/pull/613 ). And there is other more recent contributions. Where do you see any drama ?
    comment by Rexxar · 2024-04-01 · in LLaMA now goes faster on CPUs
  17. …the public cloud, developers are still targeting AVX512 when they should be targeting VNNI and AMX. https://github.com/ggerganov/llama.cpp/issues/2555
    comment by kiratp · 2024-04-01 · in LLaMA now goes faster on CPUs
  18. https://github.com/ggerganov/llama.cpp/issues/91 Justine was kicked out of llama.cpp for introducing changes before the rest of the maintainers approved them. Much drama has been had since then and I've lost interest in both projects. It's…
    comment by llm_trw · 2024-04-01 · in LLaMA now goes faster on CPUs
  19. llama.cpp probably won't be getting Jamba support anytime soon: https://github.com/ggerganov/llama.cpp/issues/6372#issuecomm... There is an MLX Mamba implementation, but nothing for Jamba either: https://github.com/alxndrTL/mamba.py/tree/ma…
    comment by lhl · 2024-03-30 · in Jamba: Production-grade Mamba-based AI model
  20. git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./server -m models/7B/ggml-model.gguf -c 2048 I don't think it'll take you the whole weekend :)
    comment by gryfft · 2024-03-27 · in Claude 3 surpasses GPT-4 on Chatbot Arena for the first time
  21. …llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/... It's also possible you're running (eg. if you're using ollama) and…
    comment by wokwokwok · 2024-03-27 · in DBRX: A new open LLM
  22. ▲ 2
    Llama.cpp: Mac Prebuilds
    story by tosh · 2024-03-22 · 0 comments
  23. ▲ 11
    Grok-1 Support for Llama.cpp
    story by schappim · 2024-03-22 · 2 comments
  24. …via 'Nitro', which was discussed here before [2]. [0] https://github.com/vllm-project/vllm [1] https://github.com/ggerganov/llama.cpp [2] https://news.ycombinator.com/item?id=38887531
    comment by lagrange77 · 2024-03-21 · in Jan: An open source alternative to ChatGPT that runs on the desktop
  25. Direct link to releases: https://github.com/ggerganov/llama.cpp/releases
    comment by schappim · 2024-03-21 · in Llama.cpp releases now ship with pre-built macOS binaries
  26. …which run on CPU but can have some layers offloaded on GPU. I definitely recommend reading the https://github.com/ggerganov/llama.cpp documentation.
    comment by fbhabbed · 2024-03-18 · in Ask HN: How to get started with local language models?
  27. If you're just looking to test it out, it's probably easiest to wait for llama.cpp to add support ( https://github.com/ggerganov/llama.cpp/issues/6120 ), and then you can run it slowly if you have enough RAM, or wait for one of the…
    comment by zone411 · 2024-03-18 · in Grok
  28. ▲ 3
    Control Vectors have been added to llama.cpp
    story by Der_Einzige · 2024-03-16 · 0 comments
  29. …a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x…
    comment by jameshart · 2024-03-15 · in Ollama now supports AMD graphics cards
  30. Here is the commit that added ROCm support to llama.cpp back in August: https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...
    comment by rahimnathwani · 2024-03-15 · in Ollama now supports AMD graphics cards
  31. …Vulkan thread for instance I see people getting it working with Polaris and even Hawaii cards. https://github.com/ggerganov/llama.cpp/pull/2059 Personally I just run it on CPU and several tokens/s is good enough for my purposes.
    comment by mysteria · 2024-03-15 · in Ollama now supports AMD graphics cards
  32. llama.cpp also supports AMD cards: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi...
    comment by Zambyte · 2024-03-15 · in Ollama now supports AMD graphics cards
  33. I'm using https://github.com/ggerganov/llama.cpp and currently mistral 7b (on a m1 macbook pro). I'm sure with some prompt examples you can get pretty good results on a smaller model. At the moment I don't have it open sourced due to it…
    comment by cheesehead · 2024-03-07 · in Show HN: Tech jobs on the command line
  34. …in GPUs. Does anyone have any idea what the bit patterns are for (starting at line 1529)? [1] https://github.com/ggerganov/llama.cpp/blob/master/ggml-cuda...
    comment by qrios · 2024-03-01 · in GGUF, the Long Way Around
  35. …it's still worth it all the way down to 2-bit. The chart in this llama.cpp PR speaks for itself: https://github.com/ggerganov/llama.cpp/pull/1684#issue-17396...
    comment by int_19h · 2024-02-29 · in The Era of 1-bit LLMs: ternary parameters for cost-effective computing
  36. These may be useful https://github.com/ggerganov/llama.cpp/blob/master/grammars https://github.com/guidance-ai/guidance?tab=readme-ov-file#c... https://github.com/eth-sri/lmql/issues/172
    comment by isaacfung · 2024-02-27 · in Show HN: Supermaven, the first code completion tool with 300k token context
  37. ▲ 17
    Gemma Is Added to Llama.cpp
    story by behnamoh · 2024-02-21 · 0 comments
  38. …just got merged, so it may take some time (could be hours or days) until this lands in ollama https://github.com/ggerganov/llama.cpp/pull/5631
    comment by SushiHippie · 2024-02-21 · in Gemma: New Open Models
  39. It should be possible to run it via llama.cpp[0] now. [0] https://github.com/ggerganov/llama.cpp/pull/5631
    comment by Kostic · 2024-02-21 · in Gemma: New Open Models
  40. If you just check out https://github.com/ggerganov/llama.cpp and run make, you’ll wind up with an executable called ‘main’ that lets you run any gguf language model you choose. Then: ./main -m ./models/30B/llama-30b.Q4_K_M.gguf --prompt…
    comment by jameshart · 2024-02-18 · in Ollama is now available on Windows in preview
  41. …will load just fine into ollama: https://old.reddit.com/r/LocalLLaMA/comments/18av9aw/quick_s... https://github.com/ggerganov/llama.cpp/discussions/2948 You can run ollama (and a web UI) pretty trivially via docker: docker run -d…
    comment by okwhateverdude · 2024-02-14 · in World model on million-length video and language with RingAttention
  42. …CUDA-compatibility layer - ZLUDA on Radeon (and not yet Intel OneAPI) - instead or in addition? https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi... https://news.ycombinator.com/item?id=38588573 What can't WebGPU abstract away…
    comment by westurner · 2024-02-12 · in AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source
  43. …- llama benchmarked on all M-series chips, and in comments there are comparisons with nvidia. https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by kolinko · 2024-02-11 · in Ask HN: Are there any reliable benchmarks for Machine Learning Model Serving?
  44. Is there a tutorial on how to get that setup running step-by-step? I only found a GitHub issue ( https://github.com/ggerganov/llama.cpp/issues/4439 ) that mentions that mainline llama.cpp isn't working for the model. Bonus question if you…
    comment by codeisawesome · 2024-02-09 · in Goody-2, the world's most responsible AI model
  45. …as well now that you could point your UI at (if you wanted to run leaner w/out ollama). https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by spmurrayzzz · 2024-02-08 · in OpenAI compatibility
  46. I recently found out about llama-cpp's official "Server" function: https://github.com/ggerganov/llama.cpp/blob/master/examples/... Works like a charm for my (simple) use case. I'm running it on an Always Free Oracle Ampere A1 instance with…
    comment by smoldesu · 2024-02-07 · in Ask HN: If you're running an LLM locally, what are you using?
  47. …is very much focused on CUDA; llama.cpp has an open issue about Metal-accelerated training: https://github.com/ggerganov/llama.cpp/issues/3799 - but no work on it so far. This is likely because training at any significant sizes requires…
    comment by int_19h · 2024-01-29 · in Meta AI releases Code Llama 70B
  48. There's https://github.com/saharNooby/rwkv.cpp , which related-ish[0] to ggml/llama.cpp [0]: https://github.com/ggerganov/llama.cpp/issues/846
    comment by kristjansson · 2024-01-29 · in Eagle 7B: Soaring past Transformers
  49. ▲ 3
    Llama.cpp supports distributed inference across machines on a local network
    story by behnamoh · 2024-01-27 · 0 comments
  50. Llama.cpp has an interactive mode, but I don't think text-generation-webui uses it. https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by sp332 · 2024-01-27 · in Brave Leo now uses Mixtral 8x7B as default