Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …or even eventually mobile phones (which are also power-sensitive due to running on batteries). [0] https://github.com/ggerganov/llama.cpp/pull/1684
    comment by kaoD · 2023-12-09 · in Paving the way to efficient architectures: StripedHyena-7B
  2. https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API
    comment by verdverm · 2023-12-08 · in JC converts the output of popular command-line tools to JSON
  3. …and control away from you. You should, instead: 1) Build a local copy of llama.cpp (literally clone https://github.com/ggerganov/llama.cpp and run 'make'). 2) Download the model version you actually want from hugging face (for example,…
    comment by wokwokwok · 2023-12-04 · in Llamafile – The easiest way to run LLMs locally on your Mac
  4. …oke/deepseek-coder-6.7B-instruct-GGUF/resolve/main/deepseek-coder-6.7b-instruct.Q5_K_M.gguf git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make ./main -ngl 32 -m ../deepseek-coder-6.7b-instruct.Q5_K_M.gguf --color -c 2048…
    comment by rightbyte · 2023-12-03 · in Run Deepseek Coder LLM locally
  5. …silicon, so I suspect M1 would be more efficient. The author recently published some benchmarks: https://github.com/ggerganov/llama.cpp/discussions/4167
    comment by tarruda · 2023-12-01 · in Are Open-Source Large Language Models Catching Up?
  6. I've been playing with various models in llama.cpp's GGUF format like this. git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make # M2 Max - 16 GB RAM wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-1…
    comment by abrinz · 2023-11-29 · in Llamafile lets you distribute and run LLMs with a single file
  7. …this could be useful. [0]: https://justine.lol/cosmopolitan/ [1]: https://www.freechat.run [2]: https://github.com/ggerganov/llama.cpp
    comment by rgbrgb · 2023-11-29 · in Llamafile lets you distribute and run LLMs with a single file
  8. …something, to emphasise the constructive nature. [1] https://github.com/outlines-dev/outlines [2] https://github.com/ggerganov/llama.cpp/pull/1773 [3] https://www.reddit.com/r/LocalLLaMA/comments/156gu8c/d_const... [4]…
    comment by darkteflon · 2023-11-29 · in How to tackle unreliability of coding assistants
  9. I'd recommended using llama.cpp and The Bloke's GGUF version of this model! https://github.com/ggerganov/llama.cpp/ https://huggingface.co/TheBloke/MonadGPT-GGUF
    comment by schmeichel · 2023-11-25 · in MonadGPT – What would have happened if ChatGPT was invented in the 17th century?
  10. hm yeah i think I need to update llama.cpp to get this fix https://github.com/ggerganov/llama.cpp/pull/3996 Thanks for trying it!
    comment by rgbrgb · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  11. …more complex than it is. It’s really is easy with llama.cpp, someone even wrote a tutorial here: https://github.com/ggerganov/llama.cpp/discussions/2948 . But yes, TheBloke tends to have conversions up very quickly as well and has made a…
    comment by paul_n · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  12. …and you have a local HTTP API. It also has a simple UI (disclaimer: to which I've contributed). https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by akx · 2023-11-22 · in LM Studio – Discover, download, and run local LLMs
  13. I've been using this [0] locally, and this model [1] on replicate. [0] https://github.com/ggerganov/llama.cpp [1] https://replicate.com/yorickvp/llava-13b/versions/6bc1c7bb0d...
    comment by xur17 · 2023-11-20 · in Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)
  14. …models so much smarter in comparison. The perplexity graph here is a pretty good illustration: https://github.com/ggerganov/llama.cpp/pull/1684 YMMV, as Mistral and Yi are not necessarily comparable like different sizes of llama, and it…
    comment by brucethemoose2 · 2023-11-19 · in StyleTTS2 – open-source Eleven-Labs-quality Text To Speech
  15. …open-source LLMs that you can run locally, on your own hardware. There are projects like llama.cpp https://github.com/ggerganov/llama.cpp and koboldcpp: https://github.com/LostRuins/koboldcpp that can make running open source models pretty…
    comment by KronisLV · 2023-11-17 · in A failed AI girlfriend product, and my lessons
  16. The llama.cpp author thinks security is "very low priority and almost unnecessary". https://github.com/ggerganov/llama.cpp/pull/651#pullrequestr... So I'm not sure why a sandbox would bundle llama.cpp and claim to be secure. They would…
    comment by jart · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  17. Not pedantic at all! https://github.com/ggerganov/llama.cpp/discussions/336 is a (somewhat rambling) discussion on whether it would even be worthwhile to use the neural engine specifically, beyond the GPU.
    comment by btown · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  18. …files are often in the range of 5GB-40GB and require a beefy GPU or a MB with 16-64GB of RAM. [1] https://github.com/ggerganov/llama.cpp
    comment by nigma · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  19. …FreeChat is 12.1 MB but the default model is ~3 GB so I'm not really losing sleep over 2 MB. [0]: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by rgbrgb · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  20. Could this be combined with something like llama.cpp's constraint-based grammar ( https://github.com/ggerganov/llama.cpp/blob/master/grammars/... ) to always enforce syntactically correct code output?
    comment by outlier99 · 2023-11-12 · in Show HN: LLM Verified with Monte Carlo Tree Search
  21. …30 min giving it a try. [1] - https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF [2] - https://github.com/ggerganov/llama.cpp
    comment by wokwokwok · 2023-11-08 · in Major outages across ChatGPT and API
  22. Mlock is an optional parameter: github.com/ggerganov/llama.cpp/tree/master/examples/main#mlock
    comment by Art9681 · 2023-11-01 · in Apple unveils M3, M3 Pro, and M3 Max
  23. …answer than GPT-4. Some of this is sampler tuning. Y'all should look at grammar based sampling ( https://github.com/ggerganov/llama.cpp/pull/1773 ) if you aren't using it already, as well as some of the "dynamic" sampling like mirostat and…
    comment by brucethemoose2 · 2023-10-31 · in Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context
  24. Please consider LLaMA.cpp ( https://github.com/ggerganov/llama.cpp ), which supports a lot of models and doesn't need an expensive GPU.
    comment by mkl · 2023-10-29 · in Show HN: BetterOCR combines and corrects multiple OCR engines with an LLM
  25. …//drivenets.com/wp-content/uploads/2023/05/blog-networ... [1] https://github.com/ornladios/ADIOS2 [2] https://github.com/ggerganov/llama.cpp [3] https://github.com/jzhang38/TinyLlama
    comment by godelski · 2023-10-21 · in What every developer should know about GPU computing
  26. Llama.cpp defaults to using metal. [0] [0] https://github.com/ggerganov/llama.cpp#metal-build
    comment by spott · 2023-10-20 · in AMD Introduces New AMD Ryzen Threadripper 7000/Pro 7000 WX
  27. I dont understand, are you claiming non-Apple devices cannot run LLMs? https://github.com/ggerganov/llama.cpp/issues/34 If you meant eGPU support, IIRC that is beta for everyone right now.
    comment by kramerger · 2023-10-19 · in LLMs up to 4x Faster With Latest NVIDIA Drivers on Windows
  28. …the developer team of a large company, it still keeps up with many of the advancements in LLMs. [0] https://github.com/ggerganov/llama.cpp
    comment by ronsor · 2023-10-17 · in SiFive Rolls Out RISC-V Cores Aimed at Generative AI and ML
  29. …grammar-based sampling [0] to ensure that the function call is at least syntactically correct. [0] https://github.com/ggerganov/llama.cpp/tree/master/grammars
    comment by huevosabio · 2023-10-16 · in MemGPT – LLMs with self-editing memory for unbounded context
  30. So, llama.cpp already somewhat supports this: https://github.com/ggerganov/llama.cpp/issues/3440
    comment by Tostino · 2023-10-09 · in Attention Sinks in LLMs for endless fluency
  31. …And then ... you'll probably could go on your own then. 1: https://github.com/tinygrad/tinygrad 2: https://github.com/ggerganov/llama.cpp
    comment by moralestapia · 2023-10-08 · in Decomposing language models into understandable components
  32. …going to be more reliable with a local model since you can mess with logits before sampling. [1] https://github.com/ggerganov/llama.cpp#constrained-output-wi... [2] https://github.com/gsuuon/ad-llama (mine)
    comment by gsuuon · 2023-10-06 · in Ask HN: Function Calling with Local LLMs?
  33. …certain rules on the output. https://github.com/Hellisotherpeople/Constrained-Text-Genera... or https://github.com/ggerganov/llama.cpp/blob/master/grammars/... for example.
    comment by sp332 · 2023-10-02 · in Ask HN: How training of LLM dedicated to code is different from LLM of “text”
  34. Yes! https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it…
    comment by brucethemoose2 · 2023-09-27 · in Mistral 7B
  35. …a training dataset of 8T tokens (/mnt/test/datasets/tokenizer_training/8T_train_data/shuffled.txt) https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
    comment by brucethemoose2 · 2023-09-27 · in Mistral 7B
  36. You'll be able to use it with llama.cpp soon [1], so, should run fine on your macbook, yes. [1]: https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
    comment by fredoliveira · 2023-09-27 · in Mistral 7B
  37. It can! TheBloke is to thank for the incredibly quick turnaround. https://github.com/ggerganov/llama.cpp/pull/3362 https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/ma...
    comment by daakus · 2023-09-27 · in Mistral 7B
  38. …and the unfortunate answer was always that the existing options were slightly complicate What about https://github.com/ggerganov/llama.cpp ? It compiles and run easily on Linux.
    comment by dorfsmay · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  39. …added ROCm linux support a little over a month ago. Details on that can be found in their readme: https://github.com/ggerganov/llama.cpp#hipblas (no luck for windows users though)
    comment by spmurrayzzz · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  40. …and behave a little differently. * https://github.com/jmorganca/ollama/blob/main/docs/api.md * https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by mgreg · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  41. It's coming to llama.cpp soon. https://github.com/ggerganov/llama.cpp/pull/3228
    comment by astrange · 2023-09-22 · in Outperforming larger language models with less training data and smaller models
  42. …use on any model. For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... Here's the one from transformers: https://github.com/huggingface/transformers/b…
    comment by dchest · 2023-09-17 · in Apple’s new Transformer-powered predictive text model
  43. …frameworks. The mostly widely accessible form of this is probably BNF grammar biasing in llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by newhouseb · 2023-09-17 · in Guidance: A guidance language for controlling large language models
  44. …Llama.cpp's Q2_K quant is 2.5625 bpw with perplexity just barely better than the next step down: https://github.com/ggerganov/llama.cpp/pull/1684 But subjectively, the Q2 quant "feels" worse than its high wikitext perplexity would suggest.…
    comment by brucethemoose2 · 2023-09-13 · in Exllamav2: Inference library for running LLMs locally on consumer-class GPUs
  45. …The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P…
    comment by jankovicsandras · 2023-09-08 · in Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI
  46. …For example, https://huggingface.co/WizardLM/WizardLM-7B-V1.0/tree/main Size with quantization, https://github.com/ggerganov/llama.cpp#quantization
    comment by notpublic · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  47. What do you mean? https://github.com/ggerganov/llama.cpp/releases/tag/master-6...
    comment by kgwgk · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  48. …If you'd like a more hands on approach, here is a manual approach to get llama running locally - https://github.com/ggerganov/llama.cpp - follow instructions to build it (note the `METAL` flag) - https://huggingface.co/models?sort=trending&…
    comment by keyle · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  49. Could this be used as a source of speculative tokens for larger llama models?, as per https://github.com/ggerganov/llama.cpp/pull/2926 Also, when are we going to start seeing open weights MOE models being released?
    comment by kristianp · 2023-09-04 · in TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
  50. https://github.com/ggerganov/llama.cpp/pull/2926
    comment by Jimmc414 · 2023-09-01 · in Full F16 precision 34B Code Llama at >20T/s on M2 Ultra