Hacker GitHub

← All repos

ggml-org/llama.cpp

View on GitHub ↗ ★ 133,134 C++

LLM inference in C/C++

738 mentions on Hacker News by 471 people: 96 stories, 642 comments · first 2023-03-10 · latest 2026-10-28

Mentions per year 738 mentions · busiest year 2023 (316)
20082026
AllStoriesComments NewestTopOldest
  1. …a training dataset of 8T tokens (/mnt/test/datasets/tokenizer_training/8T_train_data/shuffled.txt) https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
    comment by brucethemoose2 · 2023-09-27 · in Mistral 7B
  2. You'll be able to use it with llama.cpp soon [1], so, should run fine on your macbook, yes. [1]: https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
    comment by fredoliveira · 2023-09-27 · in Mistral 7B
  3. It can! TheBloke is to thank for the incredibly quick turnaround. https://github.com/ggerganov/llama.cpp/pull/3362 https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/ma...
    comment by daakus · 2023-09-27 · in Mistral 7B
  4. …and the unfortunate answer was always that the existing options were slightly complicate What about https://github.com/ggerganov/llama.cpp ? It compiles and run easily on Linux.
    comment by dorfsmay · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  5. …added ROCm linux support a little over a month ago. Details on that can be found in their readme: https://github.com/ggerganov/llama.cpp#hipblas (no luck for windows users though)
    comment by spmurrayzzz · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  6. …and behave a little differently. * https://github.com/jmorganca/ollama/blob/main/docs/api.md * https://github.com/ggerganov/llama.cpp/blob/master/examples/...
    comment by mgreg · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  7. It's coming to llama.cpp soon. https://github.com/ggerganov/llama.cpp/pull/3228
    comment by astrange · 2023-09-22 · in Outperforming larger language models with less training data and smaller models
  8. ▲ 6
    llama.cpp now supports StarCoder model series
    story by wsxiaoys · 2023-09-18 · 1 comment
  9. …use on any model. For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... Here's the one from transformers: https://github.com/huggingface/transformers/b…
    comment by dchest · 2023-09-17 · in Apple’s new Transformer-powered predictive text model
  10. …frameworks. The mostly widely accessible form of this is probably BNF grammar biasing in llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...
    comment by newhouseb · 2023-09-17 · in Guidance: A guidance language for controlling large language models
  11. …Llama.cpp's Q2_K quant is 2.5625 bpw with perplexity just barely better than the next step down: https://github.com/ggerganov/llama.cpp/pull/1684 But subjectively, the Q2 quant "feels" worse than its high wikitext perplexity would suggest.…
    comment by brucethemoose2 · 2023-09-13 · in Exllamav2: Inference library for running LLMs locally on consumer-class GPUs
  12. …The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P…
    comment by jankovicsandras · 2023-09-08 · in Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI
  13. …For example, https://huggingface.co/WizardLM/WizardLM-7B-V1.0/tree/main Size with quantization, https://github.com/ggerganov/llama.cpp#quantization
    comment by notpublic · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  14. What do you mean? https://github.com/ggerganov/llama.cpp/releases/tag/master-6...
    comment by kgwgk · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  15. …If you'd like a more hands on approach, here is a manual approach to get llama running locally - https://github.com/ggerganov/llama.cpp - follow instructions to build it (note the `METAL` flag) - https://huggingface.co/models?sort=trending&…
    comment by keyle · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  16. ▲ 4
    Llama.cpp speculative sampling: 2x faster inference for large models
    story by bobivl · 2023-09-05 · 1 comment
  17. Could this be used as a source of speculative tokens for larger llama models?, as per https://github.com/ggerganov/llama.cpp/pull/2926 Also, when are we going to start seeing open weights MOE models being released?
    comment by kristianp · 2023-09-04 · in TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens
  18. ▲ 16
    Speculative: PoC for speeding-up inference via speculative sampling by ggerganov
    story by kristianp · 2023-09-02 · 1 comment
  19. https://github.com/ggerganov/llama.cpp/pull/2926
    comment by Jimmc414 · 2023-09-01 · in Full F16 precision 34B Code Llama at >20T/s on M2 Ultra
  20. …completion and general Q/A. [0] https://huggingface.co/TheBloke/WizardCoder-Python-34B-V1.0-... [1] https://github.com/ggerganov/llama.cpp
    comment by soulofmischief · 2023-08-29 · in ChatGPT web and mobile UIs unavailable
  21. …tried running LLMs yet, here are some lines that do the job in Google Colab or locally. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf"…
    comment by jankovicsandras · 2023-08-29 · in WebLLM: Llama2 in the Browser
  22. …and local) - https://github.com/oobabooga/text-generation-webui (a web ui w/ different backends) - https://github.com/ggerganov/llama.cpp (fast local LLM runner) - https://github.com/go-skynet/LocalAI (has an openai-compatible api)
    comment by jmorgan · 2023-08-28 · in ChatGPT Enterprise
  23. …GPU, but you need to set expectations for performance. Some projects to look into are * llama.cpp - https://github.com/ggerganov/llama.cpp * KoboldCpp - https://github.com/LostRuins/koboldcpp * GPT4All - https://gpt4all.io/index.html…
    comment by loudmax · 2023-08-28 · in Continue with LocalAI: An alternative to GitHub's Copilot that runs locally
  24. …that even the best 13B quantisation would be a far cry from the 30B with 2 bit quantisation: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by pocketarc · 2023-08-26 · in Ask HN: What is the best Llama model that you can deploy on a single A10G?
  25. ▲ 2
    Llama.cpp Supports Falcon Now
    story by gslin · 2023-08-25 · 0 comments
  26. ▲ 4
    AMD ROCm Support Added to Llama.cpp
    story by irusensei · 2023-08-25 · 0 comments
  27. GGUF support was merged in llama.cpp three days ago: https://github.com/ggerganov/llama.cpp/pull/2398
    comment by w-m · 2023-08-24 · in GGUF File Format Specification
  28. …has been deprecated in favor of GGUF. https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/ggerganov/llama.cpp/pull/2398
    comment by int_19h · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  29. You can cobble together an OpenAI-esque server locally with llama.cpp: https://github.com/ggerganov/llama.cpp/issues/2766 and the script here: https://www.reddit.com/r/LocalLLaMA/comments/15ak5k4/short_g... and use Continue's GGML Model…
    comment by daemonologist · 2023-08-24 · in Show HN: Use Code Llama as Drop-In Replacement for Copilot Chat
  30. Works nearly out of the box with llama.cpp, which makes it easy to try locally: https://github.com/ggerganov/llama.cpp/issues/2766 Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt): # prints…
    comment by daemonologist · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  31. …noticeably deteriorate anything below 4 bit. You can see how much (V)RAM you need here [1]. [1] https://github.com/ggerganov/llama.cpp#quantization
    comment by syntaxing · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  32. ▲ 2
    New llama.cpp format GGUF now merged
    story by mchiang · 2023-08-21 · 0 comments
  33. ▲ 2
    GPU Support to Ggml
    story by melenaboija · 2023-08-19 · 0 comments
  34. …quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of research on this last year as well:…
    comment by lhl · 2023-08-16 · in How Is LLaMa.cpp Possible?
  35. …is lesser than you think. 5 bit quantization has negligible performance loss compared to 16 bits: https://github.com/ggerganov/llama.cpp/pull/1684
    comment by YetAnotherNick · 2023-08-16 · in How Is LLaMa.cpp Possible?
  36. …is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation: https://github.com/ggerganov/llama.cpp/pull/1684 Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B…
    comment by brucethemoose2 · 2023-08-16 · in How Is LLaMa.cpp Possible?
  37. FYI llama.cpp can do that for a "while" https://github.com/ggerganov/llama.cpp/pull/1773 Somebody is also working on a whisper.cpp version, which is maybe even more interesting because if you have grammar you can speak not only JSON but…
    comment by cztomsik · 2023-08-15 · in Show HN: LLMs can generate valid JSON 100% of the time
  38. …I think the approach using grammars (seen here, but also in things like https://github.com/ggerganov/llama.cpp/pull/1773 ) is a much more elegant solution.
    comment by simonw · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  39. …that output format guarantees are going to be important for real (non-toy) decades for LLMs [1] https://github.com/ggerganov/llama.cpp/discussions/2494
    comment by aduffy · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  40. …or https://github.com/newhouseb/clownfish or https://github.com/mkuchnik/relm or https://github.com/ggerganov/llama.cpp/pull/1773 or https://github.com/Shopify/torch-grammar Overall there are a ton of these logit based guidance systems, th
    comment by BoorishBears · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  41. …implemented grammar-based sampling last month. https://news.ycombinator.com/item?id=36819906 https://github.com/ggerganov/llama.cpp/pull/1773
    comment by sneedchucker · 2023-08-14 · in Show HN: LLMs can generate valid JSON 100% of the time
  42. ▲ 15
    Ask HN: Are you training and running custom LLMs and how are you doing it?
    story by kordlessagain · 2023-08-14 · 2 comments
  43. …you do want to self-host - there's some great libraries like https://github.com/lm-sys/FastChat and https://github.com/ggerganov/llama.cpp that might be helpful If none of these really solve your issue - feel free to email me and I'm happy…
    comment by detente18 · 2023-08-12 · in Show HN: liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching
  44. If you’re working with LLMs, just use this - https://github.com/ggerganov/llama.cpp It has Metal support.
    comment by kiratp · 2023-08-10 · in Llama 2 on ONNX runs locally
  45. How does this compare to using https://github.com/ggerganov/llama.cpp with https://huggingface.co/models?search=thebloke/llama-2-ggml ?
    comment by rrherr · 2023-08-10 · in Llama 2 on ONNX runs locally
  46. …silicon, I'd be very interested to know if there are any issues and how well it uses the processor. https://github.com/ggerganov/llama.cpp https://huggingface.co/TheBloke You should be able to at least run the 7B and probably the 13B. For…
    comment by version_five · 2023-08-09 · in Ask HN: Local LLM's
  47. I very much do appreciate your comment and will look into into llama.cpp. Was it from here: https://github.com/ggerganov/llama.cpp Do you have a guide that you followed and could link it to me or was it just from prior knowledge? Also, do…
    comment by Ms-J · 2023-08-09 · in Ask HN: Cheapest hardware to run Llama 2 70B
  48. You can also spin up the llama.cpp server ( https://github.com/ggerganov/llama.cpp/tree/master/examples/... ) and e.g. use ModelFusion to generate text, stream text, tokenize, etc. : https://modelfusion.dev/integration/model-provider/llamac…
    comment by lgrammel · 2023-08-07 · in What's new in Llama 2 and how to run it locally
  49. llama.cpp has a server with a REST API that you can use: https://github.com/ggerganov/llama.cpp/tree/master/examples/...
    comment by lgrammel · 2023-08-07 · in What's new in Llama 2 and how to run it locally
  50. …it still used my CPU... [0] https://github.com/RadeonOpenCompute/ROCm/issues/666#issueco... [1] https://github.com/ggerganov/llama.cpp/discussions/915#discu...
    comment by Kelteseth · 2023-08-07 · in Nvidia's CUDA Monopoly