Hacker GitHub

← All repos

abetlen/llama-cpp-python

View on GitHub ↗ ★ 10,667 Python

Python bindings for llama.cpp

18 mentions on Hacker News by 18 people: 4 stories, 14 comments · first 2023-04-11 · latest 2025-03-05

Mentions per year 18 mentions · busiest year 2023 (13)
20082026
AllStoriesComments NewestTopOldest
  1. …issue for a graph that has single/dual socket measurements, there is essentially no difference: https://github.com/abetlen/llama-cpp-python/issues/1098 Perhaps this is incorrect now, but I also know with 2x 4090s you don’t get higher…
    comment by buildbot · 2025-03-05 · in Apple M3 Ultra
  2. Can you use https://github.com/abetlen/llama-cpp-python or you need something ollama provide ? speaking of embeddings, you saw https://jina.ai/news/jina-embeddings-v3-a-frontier-multiling... ?
    comment by homarp · 2024-09-18 · in Knowledge graphs using Ollama and Embeddings to answer and visualizing queries
  3. ▲ 1
    Python Bindings for Llama.cpp
    story by zerojames · 2024-05-22 · 0 comments
  4. There's a Python binding for llama.cpp which is actively maintained and has worked well for me: https://github.com/abetlen/llama-cpp-python
    comment by simonw · 2024-04-29 · in Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
  5. …with JSON only for the author to write almost exactly the same code at pretty much the same time. https://github.com/abetlen/llama-cpp-python/pull/1184 But you can see how that might work. Here is the core parser code:…
    comment by luke-stanley · 2024-02-14 · in Memory and new controls for ChatGPT
  6. …wrap around it. [1] https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python
    comment by curious_cat_163 · 2023-12-10 · in Mistral-8x7B-Chat
  7. If the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.
    comment by kristianp · 2023-12-07 · in Mistral LLM on Apple Silicon Using Apple's MLX Framework Runs Instantaneously
  8. https://github.com/abetlen/llama-cpp-python#web-server
    comment by Daviey · 2023-09-07 · in Run ChatGPT-like LLMs on your laptop in 3 lines of code
  9. https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.
    comment by nodja · 2023-08-25 · in Code Llama, a state-of-the-art large language model for coding
  10. …have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though)
    comment by lhl · 2023-08-24 · in Code Llama, a state-of-the-art large language model for coding
  11. ▲ 15
    Ask HN: Are you training and running custom LLMs and how are you doing it?
    story by kordlessagain · 2023-08-14 · 2 comments
  12. …like openrouter.io. I enjoy openrouter.io myself because it supports Anthropic’s 100k models. [0]: https://github.com/abetlen/llama-cpp-python
    comment by spdustin · 2023-07-30 · in Show HN: Khoj – Chat offline with your second brain using Llama 2
  13. Thanks for sharing! We've been recommending something similar with these llama-cpp-python bindings https://github.com/abetlen/llama-cpp-python#web-server , but I'll have to check that out
    comment by sestinj · 2023-07-27 · in Show HN: Continue – Open-source coding autopilot
  14. ▲ 1
    Python Bindings for Llama.cpp
    story by EntICOnc · 2023-07-12 · 0 comments
  15. ▲ 2
    Python bindings (and OpenAI API compatible server) for llama.cpp
    story by tosh · 2023-06-04 · 1 comment
  16. I haven't tried that but https://github.com/abetlen/llama-cpp-python and https://github.com/r2d4/openlm exists
    comment by rain1 · 2023-05-14 · in Run Llama 13B with a 6GB graphics card
  17. Hi I was also looking into this and I am now using: https://github.com/abetlen/llama-cpp-python It tries to be compatible with openAI API. I managed to run AutoGPT using it (however context window is too small to be useful and even if I…
    comment by mab122 · 2023-04-18 · in GPT4All Chat – Locally-running AI chat application powered by the GPT4All-J
  18. You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with a 14 core Intel i7-9750 processor. Because…
    comment by abetlen · 2023-04-11 · in The Coming of Local LLMs