pytorch/ao
View on GitHub ↗ ★ 3,025 Python
PyTorch native quantization for training and inference
5 mentions on Hacker News by 2 people: 0 stories, 5 comments · first 2024-10-01 · latest 2025-08-21
20082026
-
…4. Related, one can play around with quantization and other "inference related" concerns (e.g. https://github.com/pytorch/ao#pytorch-native-training-to-ser... ) 5. Further related, one can play around with the fine-tuning mentioned…
comment by mdaniel · 2025-08-21 · in Gemma 3 270M re-implemented in pure PyTorch for local tinkering -
…heads * head_dim * byte_per_element * batch_size * sequence_length Some pretty charts here https://github.com/pytorch/ao/issues/539
comment by formalsystem · 2024-10-25 · in Quantized Llama models with increased speed and a reduced memory footprint -
…in python and have those algorithms run fast and you can see a lot of those algorithms here https://github.com/pytorch/ao/tree/main/torchao/prototype So for example for AWQ and GPTQ we can accelerate them by using a fast int4 kernel called…
comment by formalsystem · 2024-10-25 · in Quantized Llama models with increased speed and a reduced memory footprint -
we have experimental support for float4 training with the mx formats https://github.com/pytorch/ao/tree/main/torchao/prototype/mx... But that's waiting for Blackwell to be released so we get the hardware support. SO recommendation for now…
comment by formalsystem · 2024-10-01 · in PyTorch Native Architecture Optimization: Torchao -
…best end user facing numbers to keep things simple. We briefly hint at our contributor guide here https://github.com/pytorch/ao/issues/391 which does a tour of the APIs we provide developers implementing new algorithms But we have had…
comment by formalsystem · 2024-10-01 · in PyTorch Native Architecture Optimization: Torchao