XiongjieDai/GPU-Benchmarks-on-LLM-Inference
View on GitHub ↗ ★ 1,941 Jupyter Notebook
Multiple NVIDIA GPUs or Apple Silicon for Large Language Model Inference?
7 mentions on Hacker News by 7 people: 0 stories, 7 comments · first 2024-01-09 · latest 2026-03-09
20082026
-
A 10GB 3080 still beats even an M2 Ultra with 192GB... memory bandwidth is not the only factor. https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
comment by ranger_danger · 2026-03-09 · in How to run Qwen 3.5 locally -
For LLM inference of batch size 1, it's hard to be saturate PCIe bandwidth specially for less powerful chips. You would get close to linear performance[1]. The obvious issue is few things on multiple GPU is harder, and many softwares don't…
comment by YetAnotherNick · 2025-09-08 · in Intel Arc Pro B50 GPU Launched at $349 for Compact Workstations -
That’s what I hope for, but everything that isn’t bananas expensive with unified memory has very low memory bandwidth. DGX (Digits), Framework Desktop, and non-Ultra Macs are all around 128 gb/s, and will produce single digits tokens per…
comment by m0th87 · 2025-06-09 · in Building an AI server on a budget -
It's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
comment by revnode · 2024-12-05 · in AmpereOne: Cores Are the New MHz -
> It's still "fast enough" for even 120b models in practice Is it? This is very subjective. The Mac Studio would not be "fast enough" for me on even a 70b model, not necessarily because its output is slow, but because the prompt evaluation…
comment by angoragoats · 2024-10-29 · in New Mac Mini with M4 -
LocalLLaMA subreddit usually has some interesting benchmarks and reports. Here is one example, testing performance of different GPUs and Macs with various flavours of Llama: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
comment by abdullin · 2024-05-22 · in Ask HN: Which LLMs can run locally on most consumer computers -
https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen... should give you consumer comparison
comment by cyanydeez · 2024-01-09 · in AMD 8000G APUs launch Jan 31, $329 8700G competes with desktop GTX1650 in gaming