booydar/recurrent-memory-transformer
View on GitHub ↗ ★ 777 Jupyter Notebook
[NeurIPS 22] [AAAI 24] Recurrent Transformer-based long-context architecture.
4 mentions on Hacker News by 2 people: 1 story, 3 comments · first 2023-04-24 · latest 2023-04-24
20082026
-
…will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-repor... In addition, the group that developed FlashAttention is working on…
comment by cs702 · 2023-04-24 · in Scaling Transformer to 1M tokens and beyond with RMT -
Finetuning: https://github.com/booydar/t5-experiments/blob/scaling-repor...
comment by mysterybox · 2023-04-24 · in Code for Scaling Transformer to 1M tokens and beyond with RMT (arxiv.org) -
I found this interesting link in paper, so this can be used as ready code for RMT Transformer https://github.com/booydar/t5-experiments/tree/scaling-repor...
comment by mysterybox · 2023-04-24 · in Code for Scaling Transformer to 1M tokens and beyond with RMT (arxiv.org)