Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …since then. I can't tell if this feature request is the same thing we're talking about here? https://github.com/ggerganov/llama.cpp/issues/1552 I guess if I want more features, I should move on to something that has more features lol.
    comment by sp332 · 2023-06-25 · in What are embeddings?
  2. …local language models. [0]: https://www.reddit.com/r/LocalLLaMA/comments/14fgjqj/a_simpl... [1]: https://github.com/ggerganov/llama.cpp/discussions/1965 [2]: https://twitter.com/OfirPress
    comment by kaiokendev · 2023-06-22 · in Extending Context to 8K
  3. You can self-host an open-source model. Llama CCP is a very popular project with great docs. https://github.com/ggerganov/llama.cpp You need to be careful about liscencing - some of these models its a legal grey area whether you can use…
    comment by oli5679 · 2023-06-21 · in Show HN: Autolabel, a Python library to label and enrich text data with LLMs
  4. …about benchmarking perplexity. Here's a detailed chart of baseline FP16 vs the new k-quants: https://github.com/ggerganov/llama.cpp/pull/1684 While most evals aren't currently evaluating performance between quantized models, there are two…
    comment by lhl · 2023-06-20 · in Petaflops to the People: From Personal Compute Cluster to Person of Compute
  5. …per se but there's a lot of research and discussion going on directly in the llama.cpp repo ( https://github.com/ggerganov/llama.cpp ) if you're interested in the more technical side of things.
    comment by digitallyfree · 2023-06-18 · in OpenLLaMA 13B Released
  6. …for questioning and such. You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run
    comment by brucethemoose2 · 2023-06-18 · in OpenLLaMA 13B Released
  7. In case anyone wants to follow along: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by gardnr · 2023-06-18 · in Falcon LLM – A 40B Model
  8. Falcon doesn't work in llama.cpp yet: https://github.com/ggerganov/llama.cpp/issues/1602
    comment by gardnr · 2023-06-18 · in Falcon LLM – A 40B Model
  9. …on my old Radeon VII) which maybe someone at AMD should be supporting to make its way into master: https://github.com/ggerganov/llama.cpp/pull/1087 I'll also note that exllama merged ROCm support and it runs pretty impressively - it runs…
    comment by lhl · 2023-06-14 · in AMD Expands AI Product Lineup with GPU-Only Instinct Mi300X with 192GB Memory
  10. I just tried this [1] and it still uses my CPU even though the prompt says otherwise. [1] https://github.com/ggerganov/llama.cpp/issues/1433#issuecomm...
    comment by Kelteseth · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  11. > In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. You're comparing a single, well managed project that had put effort into user onboarding against all projects of a different…
    comment by manfre · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  12. …things to work but it's quite complex. I couldn't do it. In comparison I could just type git clone https://github.com/ggerganov/llama.cpp and make . And it worked. And since then I've managed to get llama.cpp clBLAS partial GPU…
    comment by superkuh · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  13. ▲ 728
    Llama.cpp: Full CUDA GPU Acceleration
    story by gzer0 · 2023-06-13 · 310 comments
  14. …whole change. However, the current version does use a similar approach to that which was proposed: https://github.com/ggerganov/llama.cpp/commit/f963b63afa0e05...
    comment by nl · 2023-06-12 · in Understanding DeepMind's sorting algorithm
  15. …the Llama models back in March.[1] Super impressive! These are my favorite kinds of blog posts. [1] https://github.com/ggerganov/llama.cpp/pull/613
    comment by smodad · 2023-06-12 · in Understanding DeepMind's sorting algorithm
  16. …popular belief, but the k-quants dev recently found that LLaMA 7B-65B all quantize very similarly: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by brucethemoose2 · 2023-06-10 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  17. …for CPU inference with some offload, which just got updated to a more GPTQ-like method days ago: https://github.com/ggerganov/llama.cpp/pull/1684 Some other runtimes like Apache TVM also have their own quant implementations:…
    comment by brucethemoose2 · 2023-06-10 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  18. ▲ 3
    K-Quants
    story by tosh · 2023-06-10 · 0 comments
  19. …when that was already working in the draft PR. https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 . Jart's archived comments: "my changes" "Here's how folks in the community have been reacting…
    comment by IAmNotACellist · 2023-06-08 · in GGML – AI at the Edge
  20. shrug it is in the roadmap: https://github.com/ggerganov/llama.cpp/discussions/1729 The ggml model format is growing to support more than just llama, I think.
    comment by brucethemoose2 · 2023-06-07 · in Ask HN: Suggesting better employee-supporting AI Chatbot to a mid sized corpo
  21. ▲ 1
    Llama.cpp: Roadmap June 2023
    story by tosh · 2023-06-07 · 0 comments
  22. …feel uncomfortable needing to mark their files with the initials of one. Here's the pull request: https://github.com/ggerganov/llama.cpp/pull/613
    comment by jart · 2023-06-06 · in GGML – AI at the Edge
  23. …bound to have rubbed a lot of people the wrong way. Here is the official commit undoing the change: https://github.com/ggerganov/llama.cpp/pull/711/files#diff-7...
    comment by zo1 · 2023-06-06 · in GGML – AI at the Edge
  24. …only improve from there. [1] https://old.reddit.com/r/LocalLLaMA/comments/13q6hu8/7b_perf... [2] https://github.com/ggerganov/llama.cpp/pull/1684 , https://old.reddit.com/r/LocalLLaMA/comments/141bdll/moneros...
    comment by digitallyfree · 2023-06-06 · in GGML – AI at the Edge
  25. Its not so simple anymore, see https://github.com/ggerganov/llama.cpp/pull/1684
    comment by brucethemoose2 · 2023-06-06 · in GGML – AI at the Edge
  26. https://github.com/ggerganov/llama.cpp/pull/711
    comment by pubby · 2023-06-06 · in GGML – AI at the Edge
  27. Its graph execution is still full of busyloops, e.g.: https://github.com/ggerganov/llama.cpp/blob/44f906e8537fcec9... I wonder how much more efficient it would be when Taskflow lib was used instead, or even inteltbb.
    comment by yukIttEft · 2023-06-06 · in GGML – AI at the Edge
  28. …linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/ Yeah, maybe I am an optimist, but somehow I don't see that being a big problem... > I expect LLMs to be a…
    comment by usrbinbash · 2023-06-06 · in OpenAI's plans according to Sam Altman
  29. …so LLaMA periodically resets the context and re-prompts it with a portion of its recent output. https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b1...
    comment by duskwuff · 2023-06-05 · in Apple unveils new Mac Studio and brings Apple Silicon to Mac Pro
  30. Looks like no quantized options with llama.cpp? https://github.com/ggerganov/llama.cpp/issues/1602
    comment by xrd · 2023-06-05 · in Open-Source Data Collection Platform for LLM Fine-Tuning and RLHF
  31. ▲ 4
    Full GPU Inference of LLaMA on Apple Silicon Using Metal
    story by behnamoh · 2023-06-05 · 0 comments
  32. 7B here's the linked github fwiw https://github.com/ggerganov/llama.cpp/pull/1642
    comment by jazzyjackson · 2023-06-04 · in Llama.cpp can do 40 tok/s on M2 Max, 0% CPU usage, using all 38 GPU cores
  33. …wants to see how to achieve 180+ GB/s on the M1/M2 Max, you can reproduce the methodology here: https://github.com/ggerganov/llama.cpp/pull/1642#issuecommen... I later explained the technique to write very optimized kernels for a specific…
    comment by philipturner · 2023-06-04 · in Render a neural network into CUDA/HIP code
  34. Same question, so far I've found this thread https://github.com/ggerganov/llama.cpp/issues/1602 where people work on it.
    comment by Kelamir · 2023-06-02 · in Falcon 40B LLM (which beats Llama) now Apache 2.0
  35. > https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm... Here somebody quantized it down to 29929.56MB .
    comment by Kelamir · 2023-06-02 · in Falcon 40B LLM (which beats Llama) now Apache 2.0
  36. Here is how hard it is to serve and use LLMs: https://github.com/ggerganov/llama.cpp
    comment by choppaface · 2023-06-01 · in OpenAI's plans according to sama
  37. …llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerganov/llama.cpp
    comment by samwillis · 2023-05-31 · in Ask HN: What's the best self hosted/local alternative to GPT-4?
  38. …useful to have GPU support. As a baseline, this can support anything that llama.cpp. Relevant link: https://github.com/ggerganov/llama.cpp#blas-build
    comment by ybu · 2023-05-24 · in Show HN: a desktop app to discover, download, and run offline LLMs
  39. …on https://www.reddit.com/r/LocalLLaMA . The best way for you to run the model is probably through https://github.com/ggerganov/llama.cpp . This is a plain C/C++ implementation that can run LLMs pretty efficiently. It can run the LLaMA 13B…
    comment by alpark3 · 2023-05-20 · in Ask HN: Best UNCENSORED language model comparable to ChatGPT?
  40. …implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h Are the recommendable ways to cast e.g. arrow Tensors to pytorch/tensorflow?…
    comment by westurner · 2023-05-20 · in PyTorch for WebGPU
  41. Go see yourself :) https://github.com/ggerganov/llama.cpp/blob/master/examples/... There's too many schemes right now with 4_0 and 5_1 really popular between LLM geeks.
    comment by Ambix · 2023-05-18 · in Numbers every LLM developer should know
  42. https://github.com/ggerganov/llama.cpp https://huggingface.co/mosaicml/mpt-7b
    comment by T-A · 2023-05-18 · in Show HN: Smallville – Create generative agents for simulations and games
  43. …shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
    comment by waleedk · 2023-05-17 · in Numbers every LLM developer should know
  44. Taks a look at: https://github.com/ggerganov/llama.cpp
    comment by iamflimflam1 · 2023-05-17 · in A guidance language for controlling LLMs
  45. First link: https://github.com/ggerganov/llama.cpp Which in turn has the following as the first link: https://arxiv.org/abs/2302.13971 Is it really quicker to ask here than just browse content for a bit, skimming some text or even using…
    comment by capableweb · 2023-05-14 · in Run Llama 13B with a 6GB graphics card
  46. …model architecture is slightly different, based on Pythia I guess what is needed is a pythia.cpp https://github.com/ggerganov/llama.cpp/issues/742
    comment by anentropic · 2023-05-06 · in Releasing 3B and 7B RedPajama
  47. You can do it on CPU now. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  48. Benchmarks for what you can do on CPU alone. https://github.com/ggerganov/llama.cpp/issues/34 An M1 Max does 100ms per token. A 64 core threadripper about 33ms per token.
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  49. And here are the benchmarks https://github.com/ggerganov/llama.cpp/issues/34
    comment by kiratp · 2023-05-05 · in Google “We have no moat, and neither does OpenAI”
  50. To use with llama.cpp on CPU and 8GB RAM git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build python3 -m pip install -r requirements.txt cd models && git clone…
    comment by diimdeep · 2023-05-03 · in OpenLLaMA: An Open Reproduction of LLaMA