NVIDIA/TransformerEngine
View on GitHub ↗ ★ 3,602 Python
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
3 mentions on Hacker News by 3 people: 1 story, 2 comments · first 2023-01-18 · latest 2024-09-25
20082026
-
https://github.com/NVIDIA/TransformerEngine - mixed precision FP8 is already here and provides similar accuracy to FP16/BF16
comment by eth-mld · 2023-05-23 · in In Defense of Pure 16-Bit Floating-Point Neural Networks -
…to see your benchmarks. Btw, Nvidia will be providing support for fp8 in a future release of CUDA - https://github.com/NVIDIA/TransformerEngine/issues/15 I think TMA may not matter as much for consumer cards given the disproportionate…
comment by varunkmohan · 2023-01-18 · in Which GPU(s) to get for deep learning