Hacker GitHub

← All repos

mit-han-lab/streaming-llm

View on GitHub ↗ ★ 7,276 Python

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

8 mentions on Hacker News by 7 people: 1 story, 7 comments · first 2023-10-02 · latest 2025-01-02

Mentions per year 8 mentions · busiest year 2023 (7)
20082026
AllStoriesComments NewestTopOldest
  1. https://github.com/mit-han-lab/streaming-llm On a side node, and that's what led me to the link above, I wonder if it would be possible to chain N streaming LLMs in an agent workflow and get a final output stream almost instantaneously…
    comment by Xmd5a · 2025-01-02 · in RWKV Language Model
  2. Would attention sinks work here? https://github.com/mit-han-lab/streaming-llm - it sounds like they might. In theory it doesn't involve retraining, it's just a change to how the data is managed between invocations.
    comment by regularfry · 2023-11-01 · in Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller
  3. …transform you could do to the model to avoid quite so much reprocessing per frame, but other than https://github.com/mit-han-lab/streaming-llm (which I'm not even sure directly helps) I haven't noticed anything out there.
    comment by regularfry · 2023-11-01 · in Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller
  4. See https://github.com/mit-han-lab/streaming-llm and others. There's good reason to believe that attention networks learn how to update their own weights (Forget the paper) based on their input. The attention mechanism can act like a delta…
    comment by anon291 · 2023-10-31 · in Nvidia Trains LLM on Chip Design
  5. …/notes.aimodels.fyi/llm-infinite-context-window-strea... [2] https://arxiv.org/pdf/2309.17453.pdf [3] https://github.com/mit-han-lab/streaming-llm
    comment by TheJCDenton · 2023-10-03 · in StreamingLLM: Efficient streaming technique enable infinite sequence lengths
  6. …the authors have updated the readme to add a clarified FAQ section that directly addresses this: https://github.com/mit-han-lab/streaming-llm#faq Just tested it - this definitely doesn't seem to be giving enhanced context length. It does…
    comment by bluecoconut · 2023-10-02 · in Efficient streaming language models with attention sinks
  7. The authors just uploaded a FAQ section, which may clarify some of the confusions: https://github.com/mit-han-lab/streaming-llm/blob/main/READM...
    comment by Van_Chopiszt · 2023-10-02 · in Efficient streaming language models with attention sinks