Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. Use instruct mode, with the instruction in the memory window, with the formatting discussed here: https://github.com/ggerganov/llama.cpp/issues/2262#issuecomm... If you are feeling generous, host a V2 worker using KoboldCPP. Its pretty…
    comment by brucethemoose2 · 2023-07-18 · in Llama V2 is free to try on the AI Horde
  2. some adaptation needed https://github.com/ggerganov/llama.cpp/issues/2262
    comment by swyx · 2023-07-18 · in Llama 2
  3. https://github.com/ggerganov/llama.cpp/issues/2262 Likely needs to be updated Edit: Only the case for the 34B and 70B models. 7B and 13B run as-is. You can download the GGML model already https://huggingface.co/TheBloke/Llama-2-7B-GGML…
    comment by azeirah · 2023-07-18 · in Llama 2
  4. There is an issue: https://github.com/ggerganov/llama.cpp/issues/2262 But, short story seems to be: not yet.
    comment by xrd · 2023-07-18 · in Llama 2
  5. …me to the other approach: steering the LLM's output __as it is generating tokens__ A relevant PR: https://github.com/ggerganov/llama.cpp/pull/1773 The plan is to support arbitrary grammar files to constrain token generation, similar to the…
    comment by brucethemoose2 · 2023-07-17 · in Show HN: Structured output from LLMs without reprompting
  6. …I am still figuring out the exact stack I want for productivity. I am watching this PR closely: https://github.com/ggerganov/llama.cpp/pull/1773 As well as Starcoder/Openllama finetunes for summarization, codegen and such.
    comment by brucethemoose2 · 2023-07-14 · in Ask HN: What does your AI tech stack look like?
  7. …a secret and try to pressure the government into making it illegal for you to compete with us. https://github.com/ggerganov/llama.cpp https://github.com/openlm-research/open_llama https://huggingface.co/TheBloke/open-llama-7b-open-instruct-…
    comment by PostOnce · 2023-07-11 · in GPT-4 details leaked?
  8. …accurate than gpt4j, here's to hoping we get llama.cpp support for falcon builds soon [0]! [0]: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by sodality2 · 2023-07-11 · in Show HN: Danswer – Open-source question answering across all your docs
  9. Thanks. I thought llama.cpp got CUDA capabilities a while ago? https://github.com/ggerganov/llama.cpp/pull/1827
    comment by ignoramous · 2023-07-10 · in GPT-4 API General Availability
  10. …faster than most people can type. On a $250 laptop. Isn't the future neat? the code I was running: https://github.com/ggerganov/llama.cpp and the model: https://huggingface.co/TheBloke/WizardLM-7B-uncensored-GGML There are other models…
    comment by PostOnce · 2023-07-07 · in GPT-4 API General Availability
  11. For base tooling, things like: https://huggingface.co/ (finding models and downloading them) https://github.com/ggerganov/llama.cpp (llama) https://github.com/cmp-nct/ggllm.cpp (falcon) For interactive work (art/chat/research/playing…
    comment by horsawlarway · 2023-07-07 · in ChatGPT loses users for first time, shaking faith in AI revolution
  12. I started here https://github.com/ggerganov/llama.cpp Which won't run everything, but will run model in the GGML format such as https://huggingface.co/TheBloke/llama-65B-GGML The steps are basically: 1. Download a model 2. Make sure you…
    comment by horsawlarway · 2023-07-07 · in ChatGPT loses users for first time, shaking faith in AI revolution
  13. https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke There's a LocalLLaMA subreddit, irc channels, and a whole big community around the web working on it on GitHub nd elsewhere. edit: I forgot to directly answer you: yes…
    comment by PostOnce · 2023-07-06 · in GPT-4 API General Availability
  14. …will be easy to use) versus making the setup easier but having the daily use be too hard. [0]: https://github.com/ggerganov/llama.cpp/pull/1998/commits/c19...
    comment by sebastiennight · 2023-07-06 · in LLama.cpp now has a web interface
  15. …in ggml as they started developing before #915, but I think we'll resolve this eventually. #915 - https://github.com/ggerganov/llama.cpp/discussions/915
    comment by ggerganov · 2023-07-05 · in LLama.cpp now has a web interface
  16. …Q4_K for example works out to 4.5 bits per weight instead of 4. The PR adding it has more details: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by kamilner · 2023-07-03 · in Bfloat16 support coming to Apple's Metal and PyTorch [video]
  17. There are actually two implementations being written in parallel: - https://github.com/ggerganov/llama.cpp/pull/2059 - https://github.com/ggerganov/llama.cpp/pull/2039 Ggerganov made a statement on this "redundancy" which I really like: >…
    comment by brucethemoose2 · 2023-07-01 · in WIP Llama.cpp Vulkan Implementations
  18. https://github.com/ggerganov/llama.cpp What i've been using.
    comment by linusg789 · 2023-07-01 · in Ask HN: Interesting open source LLMs you've been exploring?
  19. …since then. I can't tell if this feature request is the same thing we're talking about here? https://github.com/ggerganov/llama.cpp/issues/1552 I guess if I want more features, I should move on to something that has more features lol.
    comment by sp332 · 2023-06-25 · in What are embeddings?
  20. …local language models. [0]: https://www.reddit.com/r/LocalLLaMA/comments/14fgjqj/a_simpl... [1]: https://github.com/ggerganov/llama.cpp/discussions/1965 [2]: https://twitter.com/OfirPress
    comment by kaiokendev · 2023-06-22 · in Extending Context to 8K
  21. You can self-host an open-source model. Llama CCP is a very popular project with great docs. https://github.com/ggerganov/llama.cpp You need to be careful about liscencing - some of these models its a legal grey area whether you can use…
    comment by oli5679 · 2023-06-21 · in Show HN: Autolabel, a Python library to label and enrich text data with LLMs
  22. …about benchmarking perplexity. Here's a detailed chart of baseline FP16 vs the new k-quants: https://github.com/ggerganov/llama.cpp/pull/1684 While most evals aren't currently evaluating performance between quantized models, there are two…
    comment by lhl · 2023-06-20 · in Petaflops to the People: From Personal Compute Cluster to Person of Compute
  23. …per se but there's a lot of research and discussion going on directly in the llama.cpp repo ( https://github.com/ggerganov/llama.cpp ) if you're interested in the more technical side of things.
    comment by digitallyfree · 2023-06-18 · in OpenLLaMA 13B Released
  24. …for questioning and such. You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run
    comment by brucethemoose2 · 2023-06-18 · in OpenLLaMA 13B Released
  25. In case anyone wants to follow along: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by gardnr · 2023-06-18 · in Falcon LLM – A 40B Model
  26. Falcon doesn't work in llama.cpp yet: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by gardnr · 2023-06-18 · in Falcon LLM – A 40B Model
  27. …on my old Radeon VII) which maybe someone at AMD should be supporting to make its way into master: https://github.com/ggerganov/llama.cpp/pull/1087 I'll also note that exllama merged ROCm support and it runs pretty impressively - it runs…
    comment by lhl · 2023-06-14 · in AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory
  28. I just tried this [1] and it still uses my CPU even though the prompt says otherwise. [1] https://github.com/ggerganov/llama.cpp/issues/1433#issuecomm...
    comment by Kelteseth · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  29. > In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different…
    comment by manfre · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  30. …things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU…
    comment by superkuh · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  31. …whole change. However, the current version does use a similar approach to that which was proposed: https://github.com/ggerganov/llama.cpp/commit/f963b63afa0e05...
    comment by nl · 2023-06-12 · in Understanding DeepMind's sorting algorithm
  32. …the Llama models back in March.[1] Super impressive! These are my favorite kinds of blog posts. [1] https://github.com/ggerganov/llama.cpp/pull/613
    comment by smodad · 2023-06-12 · in Understanding DeepMind's sorting algorithm
  33. …popular belief, but the k-quants dev recently found that LLaMA 7B-65B all quantize very similarly: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by brucethemoose2 · 2023-06-10 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  34. …for CPU inference with some offload, which just got updated to a more GPTQ-like method days ago: https://github.com/ggerganov/llama.cpp/pull/1684 Some other runtimes like Apache TVM also have their own quant implementations:…
    comment by brucethemoose2 · 2023-06-10 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  35. …when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 . Jart's archived comments: "my changes" "Here's how folks in the community have been reacting…
    comment by IAmNotACellist · 2023-06-08 · in GGML – AI at the Edge
  36. shrug it is in the roadmap: https://github.com/ggerganov/llama.cpp/discussions/1729 The ggml model format is growing to support more than just llama, I think.
    comment by brucethemoose2 · 2023-06-07 · in Ask HN: Suggesting better employee-supporting AI Chatbot to a mid sized corpo
  37. …feel uncomfortable needing to mark their files with the initials of one. Here's the pull request: https://github.com/ggerganov/llama.cpp/pull/613
    comment by jart · 2023-06-06 · in GGML – AI at the Edge
  38. …bound to have rubbed a lot of people the wrong way. Here is the official commit undoing the change: https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...
    comment by zo1 · 2023-06-06 · in GGML – AI at the Edge
  39. …only improve from there. [1] https://old.reddit.com/r/LocalLLaMA/comments/13q6hu8/7b_perf... [2] https://github.com/ggerganov/llama.cpp/pull/1684 , https://old.reddit.com/r/LocalLLaMA/comments/141bdll/moneros...
    comment by digitallyfree · 2023-06-06 · in GGML – AI at the Edge
  40. Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684
    comment by brucethemoose2 · 2023-06-06 · in GGML – AI at the Edge
  41. https://github.com/ggerganov/llama.cpp/pull/711
    comment by pubby · 2023-06-06 · in GGML – AI at the Edge
  42. Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
    comment by yukIttEft · 2023-06-06 · in GGML – AI at the Edge
  43. …linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/ Yeah, maybe I am an optimist, but somehow I don't see that being a big problem... > I expect LLMs to be a…
    comment by usrbinbash · 2023-06-06 · in OpenAI's plans according to Sam Altman
  44. …so LLaMA periodically resets the context and re-prompts it with a portion of its recent output. https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b1...
    comment by duskwuff · 2023-06-05 · in Apple unveils new Mac Studio and brings Apple Silicon to Mac Pro
  45. Looks like no quantized options with llama.cpp? https://github.com/ggerganov/llama.cpp/issues/1602
    comment by xrd · 2023-06-05 · in Open-Source Data Collection Platform for LLM Fine-Tuning and RLHF
  46. 7B here's the linked github fwiw https://github.com/ggerganov/llama.cpp/pull/1642
    comment by jazzyjackson · 2023-06-04 · in Llama.cpp can do 40 tok/s on M2 Max, 0% CPU usage, using all 38 GPU cores
  47. …wants to see how to achieve 180+ GB/s on the M1/M2 Max, you can reproduce the methodology here: https://github.com/ggerganov/llama.cpp/pull/1642#issuecommen... I later explained the technique to write very optimized kernels for a specific…
    comment by philipturner · 2023-06-04 · in Render a neural network into CUDA/HIP code
  48. Same question, so far I've found this thread https://github.com/ggerganov/llama.cpp/issues/1602 where people work on it.
    comment by Kelamir · 2023-06-02 · in Falcon 40B LLM (which beats Llama) now Apache 2.0
  49. > https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm... Here somebody quantized it down to 29929.56MB .
    comment by Kelamir · 2023-06-02 · in Falcon 40B LLM (which beats Llama) now Apache 2.0
  50. Here is how hard it is to serve and use LLMs: https://github.com/ggerganov/llama.cpp
    comment by choppaface · 2023-06-01 · in OpenAI's plans according to sama