Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …that it can support is expanding [2]. Folks have also written services [3] that wrap around it. [1] https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python
    comment by curious_cat_163 · 2023-12-10 · in Mistral-8x7B-Chat
  2. what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
    comment by rgbrgb · 2023-12-10 · in Mistral-8x7B-Chat
  3. There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381
    comment by kristianp · 2023-12-10 · in Mistral-8x7B-Chat
  4. …implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h
    comment by westurner · 2023-12-10 · in Android adds ggml lib to AICore
  5. …or even eventually mobile phones (which are also power-sensitive due to running on batteries). [0] https://github.com/ggerganov/llama.cpp/pull/1684
    comment by kaoD · 2023-12-09 · in Paving the way to efficient architectures: StripedHyena-7B
  6. https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API
    comment by verdverm · 2023-12-08 · in JC converts the output of popular command-line tools to JSON
  7. …and control away from you. You should, instead: 1) Build a local copy of llama.cpp (literally clone https://github.com/ggerganov/llama.cpp and run 'make'). 2) Download the model version you actually want from hugging face (for example,…
    comment by wokwokwok · 2023-12-04 · in Llamafile – The easiest way to run LLMs locally on your Mac
  8. …oke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048…
    comment by rightbyte · 2023-12-03 · in Run Deepseek Coder LLM locally
  9. …silicon, so I suspect M1 would be more efficient. The author recently published some benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by tarruda · 2023-12-01 · in Are Open-Source Large Language Models Catching Up?
  10. I've been playing with various models in llama.cpp's GGUF format like this. git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make # M2 Max - 16 GB RAM wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-1…
    comment by abrinz · 2023-11-29 · in Llamafile lets you distribute and run LLMs with a single file
  11. …this could be useful. [0]: https://justine.lol/cosmopolitan/ [1]: https://www.freechat.run [2]: https://github.com/ggerganov/llama.cpp
    comment by rgbrgb · 2023-11-29 · in Llamafile lets you distribute and run LLMs with a single file
  12. …something, to emphasise the constructive nature. [1] https://github.com/outlines-dev/outlines [2] https://github.com/ggerganov/llama.cpp/pull/1773 [3] https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_const... [4]…
    comment by darkteflon · 2023-11-29 · in How to tackle unreliability of coding assistants
  13. ▲ 2
    Performance of Llama.cpp on Apple Silicon
    story by tosh · 2023-11-29 · 0 comments
  14. ▲ 1
    Adjust VRAM/RAM Split on Apple Silicon
    story by tosh · 2023-11-29 · 1 comment
  15. ▲ 96
    Running Llama.cpp on AWS Instances
    story by schappim · 2023-11-27 · 10 comments
  16. ▲ 2
    (2) Apple Silicon Performance · ggerganov/llama.cpp · Discussion #4167
    story by gavi · 2023-11-26 · 0 comments
  17. I'd recommended using llama.cpp and The Bloke's GGUF version of this model! https://github.com/ggerganov/llama.cpp/ https://huggingface.co/TheBloke/MonadGPT-GGUF
    comment by schmeichel · 2023-11-25 · in MonadGPT – What would have happened if ChatGPT was invented in the 17th century?
  18. hm yeah i think I need to update llama.cpp to get this fix https://github.com/ggerganov/llama.cpp/pull/3996 Thanks for trying it!
    comment by rgbrgb · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  19. …more complex than it is. It’s really is easy with llama.cpp, someone even wrote a tutorial here: https://github.com/ggerganov/llama.cpp/discussions/2948 . But yes, TheBloke tends to have conversions up very quickly as well and has made a…
    comment by paul_n · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  20. …and you have a local HTTP API. It also has a simple UI (disclaimer: to which I've contributed). https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by akx · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  21. I've been using this [0] locally, and this model [1] on replicate. [0] https://github.com/ggerganov/llama.cpp [1] https://replicate.com/yorickvp/llava-13b/versions/6bc1c7bb0d...
    comment by xur17 · 2023-11-20 · in Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)
  22. …models so much smarter in comparison. The perplexity graph here is a pretty good illustration: https://github.com/ggerganov/llama.cpp/pull/1684 YMMV, as Mistral and Yi are not necessarily comparable like different sizes of llama, and it…
    comment by brucethemoose2 · 2023-11-19 · in StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
  23. …open-source LLMs that you can run locally, on your own hardware. There are projects like llama.cpp https://github.com/ggerganov/llama.cpp and koboldcpp: https://github.com/LostRuins/koboldcpp that can make running open source models pretty…
    comment by KronisLV · 2023-11-17 · in A failed AI girlfriend product, and my lessons
  24. The llama.cpp author thinks security is "very low priority and almost unnecessary". https://github.com/ggerganov/llama.cpp/pull/651#pullrequestr... So I'm not sure why a sandbox would bundle llama.cpp and claim to be secure. They would…
    comment by jart · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  25. Not pedantic at all! https://github.com/ggerganov/llama.cpp/discussions/336 is a (somewhat rambling) discussion on whether it would even be worthwhile to use the neural engine specifically, beyond the GPU.
    comment by btown · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  26. …files are often in the range of 5GB-40GB and require a beefy GPU or a MB with 16-64GB of RAM. [1] https://github.com/ggerganov/llama.cpp
    comment by nigma · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  27. …FreeChat is 12.1 MB but the default model is ~3 GB so I'm not really losing sleep over 2 MB. [0]: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by rgbrgb · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  28. Could this be combined with something like llama.cpp's constraint-based grammar ( https://github.com/ggerganov/llama.cpp/blob/master/grammars/... ) to always enforce syntactically correct code output?
    comment by outlier99 · 2023-11-12 · in Show HN: LLM Verified with Monte Carlo Tree Search
  29. ▲ 1
    Llama on Mac M2 Ultra (Literally)
    story by behnamoh · 2023-11-10 · 0 comments
  30. …30 min giving it a try. [1] - https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF [2] - https://github.com/ggerganov/llama.cpp
    comment by wokwokwok · 2023-11-08 · in Major outages across ChatGPT and API
  31. ▲ 4
    Show HN: I built BakLLaVA and llama.cpp demo and it went viral on X
    story by Obertr · 2023-11-05 · 1 comment
  32. Mlock is an optional parameter: github.com/ggerganov/llama.cpp/tree/master/examples/main#mlock
    comment by Art9681 · 2023-11-01 · in Apple unveils M3, M3 Pro, and M3 Max
  33. …answer than GPT-4. Some of this is sampler tuning. Y'all should look at grammar based sampling ( https://github.com/ggerganov/llama.cpp/pull/1773 ) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and…
    comment by brucethemoose2 · 2023-10-31 · in Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
  34. Please consider LLaMA.cpp ( https://github.com/ggerganov/llama.cpp ), which supports a lot of models and doesn't need an expensive GPU.
    comment by mkl · 2023-10-29 · in Show HN: BetterOCR combines and corrects multiple OCR engines with an LLM
  35. …//drivenets.com/wp-content/uploads/2023/05/blog-networ... [1] https://github.com/ornladios/ADIOS2 [2] https://github.com/ggerganov/llama.cpp [3] https://github.com/jzhang38/TinyLlama
    comment by godelski · 2023-10-21 · in What every developer should know about GPU computing
  36. Llama.cpp defaults to using metal. [0] [0] https://github.com/ggerganov/llama.cpp#metal-build
    comment by spott · 2023-10-20 · in AMD Introduces New AMD Ryzen Threadripper 7000/Pro 7000 WX
  37. ▲ 2
    LLM quantization severely damages model quality and perplexity
    story by behnamoh · 2023-10-20 · 3 comments
  38. I dont understand, are you claiming non-Apple devices cannot run LLMs? https://github.com/ggerganov/llama.cpp/issues/34 If you meant eGPU support, IIRC that is beta for everyone right now.
    comment by kramerger · 2023-10-19 · in LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows
  39. …the developer team of a large company, it still keeps up with many of the advancements in LLMs. [0] https://github.com/ggerganov/llama.cpp
    comment by ronsor · 2023-10-17 · in SiFive Rolls Out RISC-V Cores Aimed at Generative AI and ML
  40. …grammar-based sampling [0] to ensure that the function call is at least syntactically correct. [0] https://github.com/ggerganov/llama.cpp/tree/master/grammars
    comment by huevosabio · 2023-10-16 · in MemGPT – LLMs with self-editing memory for unbounded context
  41. ▲ 2
    Apple predicted the rise of local LLMs, hence the M2 Ultra
    story by behnamoh · 2023-10-13 · 3 comments
  42. ▲ 8
    gg: "M2 Ultra is the absolute best personal LLM inference node you can buy."
    story by behnamoh · 2023-10-12 · 0 comments
  43. ▲ 268
    M2 Ultra can run 128 streams of Llama 2 7B in parallel
    story by behnamoh · 2023-10-11 · 173 comments
  44. ▲ 2
    Llama.cpp Was Hacked in an Evening
    story by behnamoh · 2023-10-11 · 0 comments
  45. So, llama.cpp already somewhat supports this: https://github.com/ggerganov/llama.cpp/issues/3440
    comment by Tostino · 2023-10-09 · in Attention Sinks in LLMs for endless fluency
  46. …And then ... you'll probably could go on your own then. 1: https://github.com/tinygrad/tinygrad 2: https://github.com/ggerganov/llama.cpp
    comment by moralestapia · 2023-10-08 · in Decomposing language models into understandable components
  47. …going to be more reliable with a local model since you can mess with logits before sampling. [1] https://github.com/ggerganov/llama.cpp#constrained-output-wi... [2] https://github.com/gsuuon/ad-llama (mine)
    comment by gsuuon · 2023-10-06 · in Ask HN: Function Calling with Local LLMs?
  48. …certain rules on the output. https://github.com/Hellisotherpeople/Constrained-Text-Genera... or https://github.com/ggerganov/llama.cpp/blob/master/grammars/... for example.
    comment by sp332 · 2023-10-02 · in Ask HN: How training of LLM dedicated to code is different from LLM of “text”
  49. ▲ 1
    I got llama.cpp and StarCoder – 1B to run on my P4 Retro PC
    story by vkaku · 2023-10-01 · 1 comment
  50. Yes! https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it…
    comment by brucethemoose2 · 2023-09-27 · in Mistral 7B