markasoftware/llama-cpu
View on GitHub ↗ ★ 762 Python
Fork of Facebooks LLaMa model to run on CPU
3 mentions on Hacker News by 3 people: 1 story, 2 comments · first 2023-03-04 · latest 2023-03-08
20082026
-
▲ 246
Fork of Facebook’s LLaMa model to run on CPUstory by __anon-2023__ · 2023-03-08 · 170 comments
-
Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I…
comment by downvotetruth · 2023-03-05 · in Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model -
I was able to run 7B on a CPU, inferring several words per second: https://github.com/markasoftware/llama-cpu
comment by markasoftware · 2023-03-04 · in Facebook LLAMA is being openly distributed via torrents