Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …mk600: 125t/s Llama.cpp Q5_1: 8̶4̶ 128 t/s Llama.cpp Q6_k: 7̶8̶ 116 t/s Our llama.cpp test: Build ( https://github.com/ggerganov/llama.cpp#cublas ): make -j12 LLAMA_CUBLAS=1 Run: ./main -t 16 -ngl 3̶2̶ 35 -m llama-2-7b-chat.ggmlv3.q6_K.bin…
    comment by paul_mk1 · 2023-08-06 · in MK-1
  2. …is offering, with the Q5 files here: https://huggingface.co/TheBloke/Llama-2-13B-GGML/tree/main [0] https://github.com/ggerganov/llama.cpp#quantization
    comment by lolinder · 2023-08-05 · in MK-1
  3. It's a relatively new feature, see https://github.com/ggerganov/llama.cpp/pull/1773
    comment by codethief · 2023-08-02 · in Show HN: Grammar Generator App for Llama.cpp
  4. ▲ 19
    Show HN: Grammar Generator App for Llama.cpp
    story by aduffy · 2023-08-02 · 6 comments
  5. …be really interesting for folks who don't want to send data back and forth to LLMaaS providers. https://github.com/ggerganov/llama.cpp
    comment by CharlesW · 2023-07-29 · in Show HN: LLMFlows – LangChain alternative for explicit and transparent apps
  6. …even better results using Japanese llama.cpp grammar, but I have not been down that rabbit hole: https://github.com/ggerganov/llama.cpp/pull/1773
    comment by brucethemoose2 · 2023-07-28 · in Ask HN: Are there any LLMs specialised in translating Japanese to English?
  7. …confused at this exact question too. This link might help, written by a contributor to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/1684 TLDR: Lower quantization means higher perplexity (i.e. how 'confused' the model is when…
    comment by rashidujang · 2023-07-26 · in Guide to running Llama 2 locally
  8. …llama.cpp Metal w/ large models. It will drop to CPU speeds whenever the context window is full: https://github.com/ggerganov/llama.cpp/issues/1730#issuecomm... - while sure this might be fixed in the future, it's been an issue since Metal…
    comment by lhl · 2023-07-26 · in Guide to running Llama 2 locally
  9. It has merged! https://github.com/ggerganov/llama.cpp/pull/1773 I haven't had a chance to try it yet, but I am :excitedllama:
    comment by jakedahn · 2023-07-26 · in Guide to running Llama 2 locally
  10. …uggingface.co/TheBloke/Llama-2-13B-chat-GGML/resolv... In Windows Terminal with Powershell: git clone https://github.com/ggerganov/llama.cpp cd llama.cpp mkdir build cd build cmake .. -DLLAMA_CUBLAS=ON cmake --build . --config Release cd…
    comment by shortrounddev2 · 2023-07-26 · in Guide to running Llama 2 locally
  11. …(I work at Replicate). Also, you don't need a GPU to run this script! It builds and runs llama.cpp https://github.com/ggerganov/llama.cpp
    comment by Charlieholtz · 2023-07-25 · in Guide to running Llama 2 locally
  12. …No idea if it will work, in this case, but it does with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/103
    comment by nomel · 2023-07-25 · in Guide to running Llama 2 locally
  13. According to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 , ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).
    comment by YetAnotherNick · 2023-07-25 · in LeCun: Qualcomm working with Meta to run Llama-2 on mobile devices
  14. That project already exists https://github.com/ggerganov/llama.cpp
    comment by bobbyi · 2023-07-23 · in Llama2.c: Inference llama 2 in one file of pure C
  15. I would use textsynth ( https://bellard.org/ts_server/ ) or llama.cpp ( https://github.com/ggerganov/llama.cpp ) if you're running on CPU. - I wouldn't use anything higher than a 7B model if you want decent speed. - Quantize to 4-bit to…
    comment by vikp · 2023-07-23 · in Llama2.c: Inference llama 2 in one file of pure C
  16. …based sampling, which constraints token selection to follow a rigid format like you describe. https://github.com/ggerganov/llama.cpp/pull/1773
    comment by Scaevolus · 2023-07-22 · in TypeChat
  17. At least for llama.cpp users, this recently introduced PR -- https://github.com/ggerganov/llama.cpp/pull/1773 -- introducing grammar-based sampling could potentially improve structural reliability of LLaMA output. They provide an example…
    comment by waffletower · 2023-07-21 · in TypeChat
  18. ▲ 417
    Llama: Add grammar-based sampling
    story by davepeck · 2023-07-21 · 105 comments
  19. A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773
    comment by brucethemoose2 · 2023-07-20 · in Will a prompt that enables GPT-4 to solve easy Sudoku puzzles be found?
  20. …the project (vs try to be an alternative). Ollama was originally inspired by the "server" example: https://github.com/ggerganov/llama.cpp/tree/master/examples/... This project builds on llama.cpp in a few ways: 1. Easy install! Precompiled…
    comment by jmorgan · 2023-07-20 · in Show HN: Ollama – Run LLMs on your Mac
  21. …main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook¹) ¹ https://github.com/ggerganov/llama.cpp#description
    comment by joebiden2 · 2023-07-20 · in Show HN: Ollama – Run LLMs on your Mac
  22. ▲ 3
    Llama 2: poc for running 70B on CPU
    story by tosh · 2023-07-19 · 0 comments
  23. Probably quantizing or using base weights and this project https://github.com/ggerganov/llama.cpp on a CPU machine with AVX512 instructions.
    comment by jerrygenser · 2023-07-19 · in Llama 2
  24. …to run it at home? Step 1: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma... Step 2: https://github.com/ggerganov/llama.cpp Step 3: you're welcome
    comment by ramesh31 · 2023-07-19 · in How is ChatGPT's behavior changing over time?
  25. …too much and it's brand new, so we'll see. For anyone on HN, they should probably be looking at https://github.com/ggerganov/llama.cpp and https://github.com/ggerganov/ggml directly. If you have a high-end Nvidia consumer card (3090/4090)…
    comment by lhl · 2023-07-19 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  26. …back suggesting they add a --json mode to make that easier, hasn't gained much traction though: https://github.com/ggerganov/llama.cpp/issues/1739
    comment by simonw · 2023-07-19 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  27. The gold standard of local-only model inference for LLaMA, alpaca, and friends is LLaMA-cpp, https://github.com/ggerganov/llama.cpp No dependencies, no GPU needed, just point it to a model snapshot that you download separately on…
    comment by gcr · 2023-07-18 · in Accessing Llama 2 from the command-line with the LLM-replicate plugin
  28. Use instruct mode, with the instruction in the memory window, with the formatting discussed here: https://github.com/ggerganov/llama.cpp/issues/2262#issuecomm... If you are feeling generous, host a V2 worker using KoboldCPP. Its pretty…
    comment by brucethemoose2 · 2023-07-18 · in Llama V2 is free to try on the AI Horde
  29. some adaptation needed https://github.com/ggerganov/llama.cpp/issues/2262
    comment by swyx · 2023-07-18 · in Llama 2
  30. https://github.com/ggerganov/llama.cpp/issues/2262 Likely needs to be updated Edit: Only the case for the 34B and 70B models. 7B and 13B run as-is. You can download the GGML model already https://huggingface.co/TheBloke/Llama-2-7B-GGML…
    comment by azeirah · 2023-07-18 · in Llama 2
  31. There is an issue: https://github.com/ggerganov/llama.cpp/issues/2262 But, short story seems to be: not yet.
    comment by xrd · 2023-07-18 · in Llama 2
  32. …me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the…
    comment by brucethemoose2 · 2023-07-17 · in Show HN: Structured output from LLMs without reprompting
  33. …I am still figuring out the exact stack I want for productivity. I am watching this PR closely: https://github.com/ggerganov/llama.cpp/pull/1773 As well as Starcoder/Openllama finetunes for summarization, codegen and such.
    comment by brucethemoose2 · 2023-07-14 · in Ask HN: What does your AI tech stack look like?
  34. …a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBloke/open-llama-7b-open-instruct-…
    comment by PostOnce · 2023-07-11 · in GPT-4 details leaked?
  35. …accurate than gpt4j, here's to hoping we get llama.cpp support for falcon builds soon [0]! [0]: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by sodality2 · 2023-07-11 · in Show HN: Danswer – Open-source question answering across all your docs
  36. Thanks. I thought llama.cpp got CUDA capabilities a while ago? https://github.com/ggerganov/llama.cpp/pull/1827
    comment by ignoramous · 2023-07-10 · in GPT-4 API General Availability
  37. …faster than most people can type. On a $250 laptop. Isn't the future neat? the code I was running: https://github.com/ggerganov/llama.cpp and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML There are other models…
    comment by PostOnce · 2023-07-07 · in GPT-4 API General Availability
  38. For base tooling, things like: https://huggingface.co/ (finding models and downloading them) https://github.com/ggerganov/llama.cpp (llama) https://github.com/cmp-nct/ggllm.cpp (falcon) For interactive work (art/chat/research/playing…
    comment by horsawlarway · 2023-07-07 · in ChatGPT loses users for first time, shaking faith in AI revolution
  39. I started here https://github.com/ggerganov/llama.cpp Which won't run everything, but will run model in the GGML format such as https://huggingface.co/TheBloke/llama-65B-GGML The steps are basically: 1. Download a model 2. Make sure you…
    comment by horsawlarway · 2023-07-07 · in ChatGPT loses users for first time, shaking faith in AI revolution
  40. https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke There's a LocalLLaMA subreddit, irc channels, and a whole big community around the web working on it on GitHub nd elsewhere. edit: I forgot to directly answer you: yes…
    comment by PostOnce · 2023-07-06 · in GPT-4 API General Availability
  41. …will be easy to use) versus making the setup easier but having the daily use be too hard. [0]: https://github.com/ggerganov/llama.cpp/pull/1998/commits/c19...
    comment by sebastiennight · 2023-07-06 · in LLama.cpp now has a web interface
  42. …in ggml as they started developing before #915, but I think we'll resolve this eventually. #915 - https://github.com/ggerganov/llama.cpp/discussions/915
    comment by ggerganov · 2023-07-05 · in LLama.cpp now has a web interface
  43. ▲ 328
    LLama.cpp now has a web interface
    story by xal · 2023-07-05 · 49 comments
  44. ▲ 1
    Llama.cpp now has a simple web UI for chat
    story by wsgeorge · 2023-07-04 · 0 comments
  45. …Q4_K for example works out to 4.5 bits per weight instead of 4. The PR adding it has more details: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by kamilner · 2023-07-03 · in Bfloat16 support coming to Apple's Metal and PyTorch [video]
  46. ▲ 3
    Inference at the edge: Efficient transformer model inference on-device
    story by lioeters · 2023-07-02 · 0 comments
  47. ▲ 121
    Show HN: Project S.A.T.U.R.D.A.Y. – open-source, self hosted, J.A.R.V.I.S.
    story by GRVYDEV · 2023-07-02 · 30 comments
  48. There are actually two implementations being written in parallel: - https://github.com/ggerganov/llama.cpp/pull/2059 - https://github.com/ggerganov/llama.cpp/pull/2039 Ggerganov made a statement on this "redundancy" which I really like: >…
    comment by brucethemoose2 · 2023-07-01 · in WIP Llama.cpp Vulkan Implementations
  49. ▲ 24
    WIP Llama.cpp Vulkan Implementations
    story by brucethemoose2 · 2023-07-01 · 1 comment
  50. https://github.com/ggerganov/llama.cpp What i've been using.
    comment by linusg789 · 2023-07-01 · in Ask HN: Interesting open source LLMs you've been exploring?