Hacker GitHub

← All repos

Dao-AILab/flash-attention

View on GitHub ↗ ★ 25,235 Python

Fast and memory-efficient exact attention

15 mentions on Hacker News by 14 people: 2 stories, 13 comments · first 2022-05-31 · latest 2026-03-05

Mentions per year 15 mentions · busiest year 2023 (6)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 3
    Flash Attention 4, running near matmul speeds [pdf]
    story by verdverm · 2026-03-05 · 0 comments
  2. …masked attention ("document masking"), which is so popular it's even supported by FlashAttention: https://github.com/Dao-AILab/flash-attention/issues/654 (*) Of course, the data is dirty enough that there _will_ be some duplicated stuff…
    comment by spi · 2025-12-09 · in LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
  3. …Also, flash attention is at v3-beta right now? [0] And it requires one of CUDA/Triton/ROCm? [0] https://github.com/Dao-AILab/flash-attention But maybe I'm out of the loop? Where do you see that flash attention 4 is written in Python?
    comment by davidatbu · 2025-09-06 · in ML needs a new programming language – Interview with Chris Lattner
  4. …have to only publish one , and it would work for every combination of Python and pytorch. [1] - https://github.com/Dao-AILab/flash-attention/releases/tag/v2...
    comment by kouteiheika · 2025-08-22 · in Code formatting comes to uv experimentally
  5. …it requires an H100 GPU to run 2) Flash Attention 2 added support for GQA in past version updates: https://github.com/Dao-AILab/flash-attention 3) They're comparing this implementation of Flash Attention (which is written in raw CUDA C++)…
    comment by apsec112 · 2024-07-11 · in FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision
  6. …computer to go with it, along with a PSU and some cooling. Don't know where to start with that. [1] https://github.com/Dao-AILab/flash-attention/issues/190
    comment by matroid · 2024-02-18 · in Interactive Coloring with SD ControlNet
  7. I wonder how this compares to Flash Attention ( https://github.com/HazyResearch/flash-attention ), which is the other "memory aware" Attention project I'm aware of. I guess Flash Attention is more about utilizing memory GPU SRam correctly,…
    comment by thomasahle · 2023-06-20 · in vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
  8. FlashAttention has memory linear in sequence length. https://github.com/HazyResearch/flash-attention
    comment by jph00 · 2023-05-28 · in Hacking Around ChatGPT’s Character Limits with the Code Interpreter
  9. …me of things like FNet. Otherwise, for long sequences I've yet to find something better than https://github.com/HazyResearch/flash-attention for n×n interactions and https://github.com/glassroom/heinsen_routing for n×m interactions. If…
    comment by cs702 · 2023-05-05 · in Unlimiformer: Long-Range Transformers with Unlimited Length Input
  10. …transformers is an active area of research (see e.g the great work from the Flash-attention team - https://github.com/HazyResearch/flash-attention ), and I'm sure will improve things dramatically very soon.
    comment by ofermend · 2023-05-02 · in Ask HN: Bypassing GPT-4 8k tokens limit
  11. …scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…
    comment by cs702 · 2023-04-24 · in Scaling Transformer to 1M tokens and beyond with RMT
  12. https://github.com/HazyResearch/flash-attention#memory "standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length." I guess you have just reported how many times the layer will…
    comment by GaggiX · 2023-04-19 · in StableLM: A new open-source language model
  13. …64,000 tokens on an A100. [1] https://github.com/facebookresearch/xformers/blob/main/HOWTO... [2] https://github.com/HazyResearch/flash-attention
    comment by nl · 2022-12-06 · in Turing Machines Are Recurrent Neural Networks (1996)
  14. https://github.com/HazyResearch/flash-attention
    comment by sinenomine · 2022-06-01 · in Flash Attention accelerates GPT2 3x
  15. ▲ 30
    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
    story by lnyan · 2022-05-31 · 3 comments