ryao/llama3.c
View on GitHub ↗ ★ 24 C
A fork of llama3.c used to do some R&D on inferencing
14 mentions on Hacker News by 1 person: 0 stories, 14 comments · first 2024-12-04 · latest 2025-06-15
20082026
-
…code knows that memory bandwidth is the principal bottleneck for token generation. For example: https://github.com/ryao/llama3.c
comment by ryao · 2025-06-15 · in Potential and Limitation of High-Frequency Cores and Caches (2024) -
…losses. By the way, you are lecturing someone who actually has worked on code for doing inference: https://github.com/ryao/llama3.c
comment by ryao · 2025-06-01 · in Cerebras achieves 2,500T/s on Llama 4 Maverick (400B) -
…to model size / memory bandwidth. At least, that is what I have learned from local experiments: https://github.com/ryao/llama3.c Note that prompt processing is the phase where the LLM is reading the conversation history and token…
comment by ryao · 2025-01-07 · in Nvidia announces next-gen RTX 5090 and RTX 5080 GPUs -
…AMD (since RDNA 1) both using SIMD32. To illustrate this, here are some CUDA kernels that I wrote: https://github.com/ryao/llama3.c/blob/master/rung.cu#L15 The softmax kernel for example has the hardware emulate SIMD1024, although you…
comment by ryao · 2024-12-25 · in Making AMD GPUs competitive for LLM inference (2023) -
I have been working on my own local inference software: https://github.com/ryao/llama3.c/blob/master/run.c First, CXL is useless as far as I am concerned. The smaller LLM stuff in 1 and 2 is overrated. LLMs get plenty of things wrong and…
comment by ryao · 2024-12-25 · in Making AMD GPUs competitive for LLM inference (2023) -
I have been working on inference software for my own use, with both CPU and GPU versions: https://github.com/ryao/llama3.c The only thing you wrote that makes any sense to me is “Flash attention lets you fuse the kernel”. Everything else…
comment by ryao · 2024-12-24 · in CUDA Moat Still Alive -
…I have a GEMV implementation that outperforms both the Intel MKL and OpenBLAS in my tests on Zen 3: https://github.com/ryao/llama3.c/blob/master/run.c#L429 That is unless you shoehorn GEMV into the Intel MKL's batched GEMM function, which…
comment by ryao · 2024-12-23 · in CUDA Moat Still Alive -
Ask and you shall receive: https://github.com/ryao/llama3.c/commit/dc27f2b1c6884c72bd4b... It is still a WIP, but the CUDA kernels are unlikely to see much improvement for a while as I had gone over them with a fine toothed comb for…
comment by ryao · 2024-12-19 · in Fast LLM Inference From Scratch (using CUDA) -
…precompute_input_logits() function to calculate all prompt processing via GEMM rather than GEMV: https://github.com/ryao/llama3.c/blob/master/run.c It was fairly straightforward, although the code is somewhat messy (as it is from a WIP…
comment by ryao · 2024-12-19 · in Fast LLM Inference From Scratch (using CUDA) -
…it. I have another fork here that uses the MKL to make the prompt processing portion run faster: https://github.com/ryao/llama3.c When I am happy with the CUDA version, I will either push it to a branch or to its own set of files in master.
comment by ryao · 2024-12-16 · in Fast LLM Inference From Scratch (using CUDA) -
…I have been working on doing inference on a Ryzen 7 5800X lately and I have had good results: https://github.com/ryao/llama3.c/blob/master/run.c Running on a GPU like my 3090 Ti will likely outperform it by two orders of magnitude, but I…
comment by ryao · 2024-12-04 · in Intel announces Arc B-series "Battlemage" discrete graphics with Linux support -
…some local llama 3 inference software on a machine with 3200MT/s ECC RAM and a Ryzen 7 5800X: https://github.com/ryao/llama3.c My experience is that input processing (prompt processing) is compute bottlenecked in GEMM. AVX-512 would help…
comment by ryao · 2024-12-04 · in Intel announces Arc B-series "Battlemage" discrete graphics with Linux support -
…is much more simple than you think: See the precompute_input_logits() and forward() functions here: https://github.com/ryao/llama3.c/blob/master/run.c#L520 As a preface, precompute_input_logits() is really just a generalized version of the…
comment by ryao · 2024-12-04 · in Intel announces Arc B-series "Battlemage" discrete graphics with Linux support -
…CPU, but I have been thinking about how I would port it to a GPU) and would like to do a rebuttal: https://github.com/ryao/llama3.c Inference workloads are easy to parallelize to N cards with minimal connectivity between them. The Nvlink…
comment by ryao · 2024-12-04 · in Intel announces Arc B-series "Battlemage" discrete graphics with Linux support