Hacker GitHub

← All repos

glassroom/sata_attention

View on GitHub ↗ ★ 36 Python

Reference implementation of "Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation" (Heinsen and Kozachkov, 2026)

4 mentions on Hacker News by 3 people: 0 stories, 4 comments · first 2026-02-04 · latest 2026-02-07

Mentions per year 4 mentions · busiest year 2026 (4)
20082026
AllStoriesComments NewestTopOldest
  1. …toy example is with 8 Taylor terms, applied to a context with 1B tokens, each token with 1K heads: https://github.com/glassroom/sata_attention That toy example is not practical with the quadratic formulation, because it would require…
    comment by fheinsen · 2026-02-07 · in Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
  2. …order per layer, per head, etc. The github repository lists many other possible improvements: https://github.com/glassroom/sata_attention#proof-of-concept
    comment by fheinsen · 2026-02-04 · in Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
  3. Reference implementation: https://github.com/glassroom/sata_attention
    comment by NedCode · 2026-02-04 · in Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation
  4. Github here: https://github.com/glassroom/sata_attention
    comment by spacewhales · 2026-02-04 · in Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation