Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …completion and general Q/A. [0] https://huggingface.co/TheBloke/WizardCoder-Python-34B-V1.0-... [1] https://github.com/ggerganov/llama.cpp
    comment by soulofmischief · 2023-08-29 · in ChatGPT web and mobile UIs unavailable
  2. …tried running LLMs yet, here are some lines that do the job in Google Colab or locally. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf"…
    comment by jankovicsandras · 2023-08-29 · in WebLLM: Llama2 in the Browser
  3. …and local) - https://github.com/oobabooga/text-generation-webui (a web ui w/ different backends) - https://github.com/ggerganov/llama.cpp (fast local LLM runner) - https://github.com/go-skynet/LocalAI (has an openai-compatible api)
    comment by jmorgan · 2023-08-28 · in ChatGPT Enterprise
  4. …GPU, but you need to set expectations for performance. Some projects to look into are * llama.cpp - https://github.com/ggerganov/llama.cpp * KoboldCpp - https://github.com/LostRuins/koboldcpp * GPT4All - https://gpt4all.io/index.html…
    comment by loudmax · 2023-08-28 · in Continue with LocalAI: An alternative to GitHub's Copilot that runs locally
  5. …that even the best 13B quantisation would be a far cry from the 30B with 2 bit quantisation: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by pocketarc · 2023-08-26 · in Ask HN: What is the best Llama model that you can deploy on a single A10G?
  6. GGUF support was merged in llama.cpp three days ago: https://github.com/ggerganov/llama.cpp/pull/2398
    comment by w-m · 2023-08-24 · in GGUF File Format Specification
  7. …has been deprecated in favor of GGUF. https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/ggerganov/llama.cpp/pull/2398
    comment by int_19h · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  8. You can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 and the script here: https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g... and use Continue's GGML Model…
    comment by daemonologist · 2023-08-24 · in Show HN: Use Code Llama as Drop-In Replacement for Copilot Chat
  9. Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints…
    comment by daemonologist · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  10. …noticeably deteriorate anything below 4 bit. You can see how much (V)RAM you need here [1]. [1] https://github.com/ggerganov/llama.cpp#quantization
    comment by syntaxing · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  11. …quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of research on this last year as well:…
    comment by lhl · 2023-08-16 · in How Is LLaMa.cpp Possible?
  12. …is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by YetAnotherNick · 2023-08-16 · in How Is LLaMa.cpp Possible?
  13. …is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation: https://github.com/ggerganov/llama.cpp/pull/1684 Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B…
    comment by brucethemoose2 · 2023-08-16 · in How Is LLaMa.cpp Possible?
  14. FYI llama.cpp can do that for a "while" https://github.com/ggerganov/llama.cpp/pull/1773 Somebody is also working on a whisper.cpp version, which is maybe even more interesting because if you have grammar you can speak not only JSON but…
    comment by cztomsik · 2023-08-15 · in Show HN: LLMs can generate valid JSON 100% of the time
  15. …I think the approach using grammars (seen here, but also in things like https://github.com/ggerganov/llama.cpp/pull/1773 ) is a much more elegant solution.
    comment by simonw · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  16. …that output format guarantees are going to be important for real (non-toy) decades for LLMs [1] https://github.com/ggerganov/llama.cpp/discussions/2494
    comment by aduffy · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  17. …or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or https://github.com/ggerganov/llama.cpp/pull/1773 or https://github.com/Shopify/torch-grammar Overall there are a ton of these logit based guidance systems, th
    comment by BoorishBears · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  18. …implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773
    comment by sneedchucker · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  19. …you do want to self-host - there's some great libraries like https://github.com/lm-sys/FastChat and https://github.com/ggerganov/llama.cpp that might be helpful If none of these really solve your issue - feel free to email me and I'm happy…
    comment by detente18 · 2023-08-12 · in Show HN: liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching
  20. If you’re working with LLMs, just use this - https://github.com/ggerganov/llama.cpp It has Metal support.
    comment by kiratp · 2023-08-10 · in Llama 2 on ONNX runs locally
  21. How does this compare to using https://github.com/ggerganov/llama.cpp with https://huggingface.co/models?search=thebloke/llama-2-ggml ?
    comment by rrherr · 2023-08-10 · in Llama 2 on ONNX runs locally
  22. …silicon, I'd be very interested to know if there are any issues and how well it uses the processor. https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke You should be able to at least run the 7B and probably the 13B. For…
    comment by version_five · 2023-08-09 · in Ask HN: Local LLM's
  23. I very much do appreciate your comment and will look into into llama.cpp. Was it from here: https://github.com/ggerganov/llama.cpp Do you have a guide that you followed and could link it to me or was it just from prior knowledge? Also, do…
    comment by Ms-J · 2023-08-09 · in Ask HN: Cheapest hardware to run Llama 2 70B
  24. You can also spin up the llama.cpp server ( https://github.com/ggerganov/llama.cpp/tree/master/examples/... ) and e.g. use ModelFusion to generate text, stream text, tokenize, etc. : https://modelfusion.dev/integration/model-provider/llamac…
    comment by lgrammel · 2023-08-07 · in What's new in Llama 2 and how to run it locally
  25. llama.cpp has a server with a REST API that you can use: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by lgrammel · 2023-08-07 · in What's new in Llama 2 and how to run it locally
  26. …it still used my CPU... [0] https://github.com/RadeonOpenCompute/ROCm/issues/666#issueco... [1] https://github.com/ggerganov/llama.cpp/discussions/915#discu...
    comment by Kelteseth · 2023-08-07 · in Nvidia's CUDA Monopoly
  27. …mk600: 125t/s Llama.cpp Q5_1: 8̶4̶ 128 t/s Llama.cpp Q6_k: 7̶8̶ 116 t/s Our llama.cpp test: Build ( https://github.com/ggerganov/llama.cpp#cublas ): make -j12 LLAMA_CUBLAS=1 Run: ./main -t 16 -ngl 3̶2̶ 35 -m llama-2-7b-chat.ggmlv3.q6_K.bin…
    comment by paul_mk1 · 2023-08-06 · in MK-1
  28. …is offering, with the Q5 files here: https://huggingface.co/TheBloke/Llama-2-13B-GGML/tree/main [0] https://github.com/ggerganov/llama.cpp#quantization
    comment by lolinder · 2023-08-05 · in MK-1
  29. It's a relatively new feature, see https://github.com/ggerganov/llama.cpp/pull/1773
    comment by codethief · 2023-08-02 · in Show HN: Grammar Generator App for Llama.cpp
  30. …be really interesting for folks who don't want to send data back and forth to LLMaaS providers. https://github.com/ggerganov/llama.cpp
    comment by CharlesW · 2023-07-29 · in Show HN: LLMFlows – LangChain alternative for explicit and transparent apps
  31. …even better results using Japanese llama.cpp grammar, but I have not been down that rabbit hole: https://github.com/ggerganov/llama.cpp/pull/1773
    comment by brucethemoose2 · 2023-07-28 · in Ask HN: Are there any LLMs specialised in translating Japanese to English?
  32. …confused at this exact question too. This link might help, written by a contributor to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/1684 TLDR: Lower quantization means higher perplexity (i.e. how 'confused' the model is when…
    comment by rashidujang · 2023-07-26 · in Guide to running Llama 2 locally
  33. …llama.cpp Metal w/ large models. It will drop to CPU speeds whenever the context window is full: https://github.com/ggerganov/llama.cpp/issues/1730#issuecomm... - while sure this might be fixed in the future, it's been an issue since Metal…
    comment by lhl · 2023-07-26 · in Guide to running Llama 2 locally
  34. It has merged! https://github.com/ggerganov/llama.cpp/pull/1773 I haven't had a chance to try it yet, but I am :excitedllama:
    comment by jakedahn · 2023-07-26 · in Guide to running Llama 2 locally
  35. …uggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolv... In Windows Terminal with Powershell: git clone https://github.com/ggerganov/llama.cpp cd llama.cpp mkdir build cd build cmake .. -DLLAMA_CUBLAS=ON cmake --build . --config Release cd…
    comment by shortrounddev2 · 2023-07-26 · in Guide to running Llama 2 locally
  36. …(I work at Replicate). Also, you don't need a GPU to run this script! It builds and runs llama.cpp https://github.com/ggerganov/llama.cpp
    comment by Charlieholtz · 2023-07-25 · in Guide to running Llama 2 locally
  37. …No idea if it will work, in this case, but it does with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/103
    comment by nomel · 2023-07-25 · in Guide to running Llama 2 locally
  38. According to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 , ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).
    comment by YetAnotherNick · 2023-07-25 · in LeCun: Qualcomm working with Meta to run Llama-2 on mobile devices
  39. That project already exists https://github.com/ggerganov/llama.cpp
    comment by bobbyi · 2023-07-23 · in Llama2.c: Inference llama 2 in one file of pure C
  40. I would use textsynth ( https://bellard.org/ts_server/ ) or llama.cpp ( https://github.com/ggerganov/llama.cpp ) if you're running on CPU. - I wouldn't use anything higher than a 7B model if you want decent speed. - Quantize to 4-bit to…
    comment by vikp · 2023-07-23 · in Llama2.c: Inference llama 2 in one file of pure C
  41. …based sampling, which constraints token selection to follow a rigid format like you describe. https://github.com/ggerganov/llama.cpp/pull/1773
    comment by Scaevolus · 2023-07-22 · in TypeChat
  42. At least for llama.cpp users, this recently introduced PR -- https://github.com/ggerganov/llama.cpp/pull/1773 -- introducing grammar-based sampling could potentially improve structural reliability of LLaMA output. They provide an example…
    comment by waffletower · 2023-07-21 · in TypeChat
  43. A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773
    comment by brucethemoose2 · 2023-07-20 · in Will a prompt that enables GPT-4 to solve easy Sudoku puzzles be found?
  44. …the project (vs try to be an alternative). Ollama was originally inspired by the "server" example: https://github.com/ggerganov/llama.cpp/tree/master/examples/... This project builds on llama.cpp in a few ways: 1. Easy install! Precompiled…
    comment by jmorgan · 2023-07-20 · in Show HN: Ollama – Run LLMs on your Mac
  45. …main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook¹) ¹ https://github.com/ggerganov/llama.cpp#description
    comment by joebiden2 · 2023-07-20 · in Show HN: Ollama – Run LLMs on your Mac
  46. Probably quantizing or using base weights and this project https://github.com/ggerganov/llama.cpp on a CPU machine with AVX512 instructions.
    comment by jerrygenser · 2023-07-19 · in Llama 2
  47. …to run it at home? Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma... Step 2: https://github.com/ggerganov/llama.cpp Step 3: you're welcome
    comment by ramesh31 · 2023-07-19 · in How is ChatGPT's behavior changing over time?
  48. …too much and it's brand new, so we'll see. For anyone on HN, they should probably be looking at https://github.com/ggerganov/llama.cpp and https://github.com/ggerganov/ggml directly. If you have a high-end Nvidia consumer card (3090/4090)…
    comment by lhl · 2023-07-19 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  49. …back suggesting they add a --json mode to make that easier, hasn't gained much traction though: https://github.com/ggerganov/llama.cpp/issues/1739
    comment by simonw · 2023-07-19 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  50. The gold standard of local-only model inference for LLaMA, alpaca, and friends is LLaMA-cpp, https://github.com/ggerganov/llama.cpp No dependencies, no GPU needed, just point it to a model snapshot that you download separately on…
    comment by gcr · 2023-07-18 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin