mozilla-ai/llamafile on Hacker News

See llamafile ( https://github.com/Mozilla-Ocho/llamafile ), a standalone packaging of llama.cpp that runs an LLM locally. It will use the GPU, but falls back on the CPU. CPU-only performance of small, quantized models is still pretty…

comment by ynniv · 2024-05-22 · in Ask HN: Which LLMs can run locally on most consumer computers

I developed this originally for llamafile, which was included in the last two releases: https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.8.2 Now we're upstreaming it to the llama.cpp project. There are other performance…

comment by jart · 2024-05-15 · in New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy

▲ 3

Llamafile 0.8.2 Release with Embedding Subcmd in CLI and Performance Boost

story by lijunhao · 2024-05-10 · 0 comments

I think llamafile supports openai compatible api.. https://github.com/Mozilla-Ocho/llamafile

comment by indigodaddy · 2024-05-01 · in Show HN: I'm 16 and building an AI based startup called Factful with friends

▲ 4

llamafile v0.8

story by birriel · 2024-04-25 · 0 comments

…the ecosystem, but I think you should be able to run this new model as is against a llamafile? [0] https://github.com/Mozilla-Ocho/llamafile

comment by fbdab103 · 2024-04-10 · in Mistral AI Launches New 8x22B MOE Model

…(HA). Maybe someone is working on this already? TTS: https://github.com/SYSTRAN/faster-whisper LLM: https://github.com/Mozilla-Ocho/llamafile/releases LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D... HA:…

comment by thangalin · 2024-04-04 · in Apple explores home robotics as potential 'next big thing'

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...…

comment by Dobiasd · 2024-04-04 · in LLaMA now goes faster on CPUs

Any performance benchmark against 'llamafile'[0] or others? [0] - https://github.com/mozilla-Ocho/llamafile

comment by DrNosferatu · 2024-04-03 · in PyTorch Library for Running LLM on Intel CPU and GPU

Ollama https://ollama.com/ or llamafile https://github.com/Mozilla-Ocho/llamafile *there are some tools/projects already integrating it that way

comment by throwaway888abc · 2024-04-01 · in Ask HN: CPU-based LLM for Markdown editor

…sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register" https://github.com/mozilla-ocho/llamafile/?tab=readme-ov-fil...

comment by jart · 2024-04-01 · in LLaMA now goes faster on CPUs

…a binary with a few command-line options, seems pretty close to what you describe you want. [1] https://github.com/Mozilla-Ocho/llamafile

comment by TachyonicBytes · 2024-04-01 · in Ask HN: What is the current (Apr. 2024) gold standard of running an LLM locally?

…the README "gotcha" section). Solved my running "assimilate" [3] on the downloaded llamafile. [1] https://github.com/Mozilla-Ocho/llamafile/ [2] https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-i…

comment by yaantc · 2024-04-01 · in LLaMA now goes faster on CPUs

…will also benefit. Though of course everyone should also look directly at releases from llamafile https://github.com/mozilla-Ocho/llamafile .

comment by dagaci · 2024-04-01 · in LLaMA now goes faster on CPUs

…also happen to be actually portable executables, ehm, excuse me, αcτµαlly pδrταblε εxεcµταblες. https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

comment by formerly_proven · 2024-03-31 · in Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times for AMD Zen 4

…file size limit. For Windows users, here's an example for the Mistral LLM: curl -L -o llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/... curl -L -o mistral.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-…

comment by supposemaybe · 2024-03-31 · in Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times for AMD Zen 4

…giving credit and can continue having a nice single file binary. The code which does that is here: https://github.com/Mozilla-Ocho/llamafile/blob/9c53c272650bb... So let me install the latest Ollama on my Windows computer and see if…

comment by jart · 2024-03-16 · in Ollama now supports AMD graphics cards

…don't avoid it altogether, and they certainly don't seem to think its unnecessary. (source: https://github.com/Mozilla-Ocho/llamafile ) - 2nd sentence in README: "Our goal is to make open LLMs much more accessible to both developers and…

comment by refulgentis · 2024-03-16 · in Ollama now supports AMD graphics cards

…would be cool to also discuss how llamafiles[1] work, and how they differ from the GGUF files. [1]: https://github.com/Mozilla-Ocho/llamafile

comment by mc10 · 2024-03-01 · in GGUF, the Long Way Around

Not as far as I know, but llamafiles are almost easier to get running than native packages. https://github.com/Mozilla-Ocho/llamafile

comment by louismerlin · 2024-03-01 · in Ask HN: What AI assistants are already bundled for Linux?

have you seen llamafile[0]? [0] https://github.com/Mozilla-Ocho/llamafile

comment by vorticalbox · 2024-02-28 · in Show HN: I made an app to use local AI as daily driver

…/ Command line (not for me): - https://ollama.com - https://privategpt.dev - https://serge.chat - https://github.com/Mozilla-Ocho/llamafile Pending to check: - https://recurse.chat Feel free to recommend more!

comment by pentagrama · 2024-02-28 · in Show HN: I made an app to use local AI as daily driver

…-p '[INST]Write a story about llamas[/INST]' [0] https://llm.datasette.io/en/stable/index.html [1] https://github.com/Mozilla-Ocho/llamafile Edit: added llm quickstart from the intro page

comment by fbdab103 · 2024-02-27 · in Ask HN: People who switched from GPT to their own models. How was it?

…use llamafile for this. It is a standalone executable that is portable across most popular OSes. https://github.com/Mozilla-Ocho/llamafile/releases So, download the executable from the releases page under assets. You want either just main…

comment by superkuh · 2024-02-23 · in Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

So... llamafile release? https://github.com/Mozilla-Ocho/llamafile

comment by dartharva · 2024-02-23 · in Gemma.cpp: lightweight, standalone C++ inference engine for Gemma models

Can we have llamafile releases as well? https://github.com/Mozilla-Ocho/llamafile

comment by dartharva · 2024-02-21 · in Gemma: New Open Models

…gpu support. on windows, it only depends on the graphics driver, thanks to our tinyBLAS library. https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.6.2 By default it opens a browser tab with a chat gui. You can run it as a cli…

comment by jart · 2024-02-17 · in Ollama is now available on Windows in preview

…for locally hosting LLMs over the last few months have been amazing. I was ranting about how easy https://github.com/Mozilla-Ocho/llamafile is just a few hours ago [1]. Now I'm torn as to which one to use :) 1: Quite literally hours ago:…

comment by ultrasaurus · 2024-02-08 · in OpenAI compatibility

Slightly off topic, here is the best local llama.cpp wrapper I've run into: https://github.com/Mozilla-Ocho/llamafile You can download any .gguf model (not just the ones in their examples) and run it locally (as long as you have the ram…

comment by depingus · 2024-02-08 · in Localllm lets you develop gen AI apps on local CPUs

Have you looked at llamafile[0]? It can spin up a openAi compatible API for a while host of LLM. [0] https://github.com/Mozilla-Ocho/llamafile

comment by vorticalbox · 2024-02-07 · in Oslo acquires ChatGPT for 110k students and teachers

…collection, I recognise the output style the template-feeling elements of it, much more now. [1] https://github.com/Mozilla-Ocho/llamafile/ [2] https://news.ycombinator.com/item?id=19251755

comment by jodrellblank · 2024-02-01 · in Show HN: Some blind hackers are bridging IRC to LMMs running locally

Does anyone have any information about LLaVA-1.6 being bundled as a LLamafile? https://github.com/Mozilla-Ocho/llamafile

comment by bsenftner · 2024-02-01 · in LLaVA-1.6: Improved reasoning, OCR, and world knowledge

In my experience award for easiest to run locally will go to llamafile models https://github.com/Mozilla-Ocho/llamafile .

comment by acd10j · 2024-01-25 · in Ollama releases Python and JavaScript Libraries

…one executable LLM that works on Mac, windows and Linux. Currently pushing for application note https://github.com/Mozilla-Ocho/llamafile/pull/178 to encourage integration. Would be good to hear your thoughts on making it easier for home…

comment by mofosyne · 2024-01-14 · in Building a fully local LLM voice assistant to control my smart home

…tools that require more setup than downloading and running a single llamafile executable? [1] https://github.com/Mozilla-Ocho/llamafile/

comment by barnabee · 2024-01-09 · in Mixtral 8x7B: A sparse Mixture of Experts language model

…[0] https://hacks.mozilla.org/2023/11/introducing-llamafile/ [1] https://github.com/Mozilla-Ocho/llamafile#quickstart

comment by noman-land · 2024-01-09 · in Mixtral 8x7B: A sparse Mixture of Experts language model

You can download the llamafile executables from Mozilla's release page here: https://github.com/Mozilla-Ocho/llamafile/releases and then use the `-m` flag which lets you load any GGUF weights you want from Hugging Face. A lot of people I…

comment by jart · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs

….args You'd then grab the latest llamafile release binary off https://github.com/Mozilla-Ocho/llamafile/releases/ along with our zipalign program, and use it to insert the weights back into the new file: zipalign -j0 lla

comment by jart · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs

…somehow encapsulates the model and the code used to interface with it. Is it the one listed here? https://github.com/Mozilla-Ocho/llamafile

comment by spit2wind · 2023-12-31 · in Emacs-copilot: Large language model code completion for Emacs

Mixtral-8x7B-Instruct is also available in a llamafile now: https://github.com/Mozilla-Ocho/llamafile#other-example-llam...

comment by throwup238 · 2023-12-18 · in Many options for running Mistral models in your terminal using LLM

Try llamafile https://github.com/Mozilla-Ocho/llamafile I have Mistral 7B running this way on a 10 year old laptop and it only seems to use a few GB with it's memory mapping approach.

comment by trailbits · 2023-12-17 · in Run Mistral 7B on M1 Mac

▲ 6

Llamafile 0.4 now with Mixtral support

story by louismerlin · 2023-12-14 · 1 comment

Somehow wound up installing from https://github.com/mozilla-Ocho/llamafile Don't ask me how I wound up there - I apparently followed links from this HN article.

comment by CliffStoll · 2023-12-13 · in Bash one-liners for LLMs

…a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82

comment by jart · 2023-12-13 · in Bash one-liners for LLMs

…to do it. I don't know how helpful the model is with screenshots though. 1. Download LLaVA from https://github.com/Mozilla-Ocho/llamafile 2. Run Whisper locally for speech to text 3. Save screenshots and send to the model, with a script…

comment by dave1010uk · 2023-12-12 · in Show HN: Open-source macOS AI copilot using vision and voice

…right now. Weights here: https://huggingface.co/jartine/llava-v1.5-7B-GGUF/tree/main Sauce here: https://github.com/mozilla-Ocho/llamafile

comment by jart · 2023-12-11 · in Show HN: I Remade the Fake Google Gemini Demo, Except Using GPT-4 and It's Real

I’ve been using llava via https://github.com/Mozilla-Ocho/llamafile which runs on any modern system.

comment by ok_dad · 2023-12-11 · in Show HN: I Remade the Fake Google Gemini Demo, Except Using GPT-4 and It's Real

Closing the loop for anyone reading this thread -- see https://github.com/Mozilla-Ocho/llamafile/issues/11 for the fix. Thanks jart!

comment by derwiki · 2023-12-01 · in Llamafile lets you distribute and run LLMs with a single file

Looks like this may be due to crowdstrike, which I also have installed on this machine: https://github.com/Mozilla-Ocho/llamafile/issues/14#issuecom...

comment by carbocation · 2023-11-30 · in Llamafile lets you distribute and run LLMs with a single file

We have an issue here tracking this: https://github.com/Mozilla-Ocho/llamafile/issues/14 Please follow that issue for updates.

comment by jart · 2023-11-30 · in Llamafile lets you distribute and run LLMs with a single file