glassroom/heinsen_routing
View on GitHub ↗ ★ 171 Python
Reference implementation of "An Algorithm for Routing Vectors in Sequences" (Heinsen, 2022) and "An Algorithm for Routing Capsules in All Domains" (Heinsen, 2019), for composing deep neural networks.
14 mentions on Hacker News by 3 people: 5 stories, 9 comments · first 2019-10-24 · latest 2025-07-15
20082026
-
…on HN! --- EDIT: The hierarchical composition and routing aspects of this work vaguely remind me of https://github.com/glassroom/heinsen_routing/ but it has been a while since I played with that. UPDATE: After spending a bit more time on…
comment by cs702 · 2025-07-15 · in Hierarchical Modeling (H-Nets) -
…HN. I've added it to my reading list. EDIT: I'm reminded of this other type of associative memory: https://github.com/glassroom/heinsen_routing . The idea there is to compute a mixture of memories that best predicts the given input…
comment by cs702 · 2025-01-13 · in Titans: Learning to Memorize at Test Time -
…m/lucidrains/memorizing-transformers-pytorc... ( great work by lucidrains) or via routed queries with https://github.com/glassroom/heinsen_routing (don't fully understand it; apparently related to attention). Both approaches seemed to…
comment by cs702 · 2024-04-27 · in What can LLMs never do? -
…cosine similarity. Nice. I like it and expect it will work well in many scenarios. Also check out https://github.com/glassroom/heinsen_routing . It takes N embeddings and outputs M embeddings (instead of one), and can optionally give you…
comment by cs702 · 2023-09-12 · in A Surprisingly Effective Way to Estimate Token Importance in LLM Prompts -
…find something better than https://github.com/HazyResearch/flash-attention for n×n interactions and https://github.com/glassroom/heinsen_routing for n×m interactions. If anyone has other suggestions, I'd love to hear about them.
comment by cs702 · 2023-05-05 · in Unlimiformer: Long-Range Transformers with Unlimited Length Input -
…experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing * RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in…
comment by cs702 · 2023-04-24 · in Scaling Transformer to 1M tokens and beyond with RMT -
In principle, yes, you can, by using something like this on the last layer https://github.com/glassroom/heinsen_routing#sequence-to-vec... ... but keep in mind it will not match the training efficiency of triangular masking in…
comment by cs702 · 2023-04-11 · in From deep to long learning? -
…algorithm, related to self-attention, that can handle sequences with 1M+ tokens in a single GPU: https://github.com/glassroom/heinsen_routing . Right now, you can take 1,000 sequences of hidden states computed by a pretrained transformer,…
comment by cs702 · 2023-04-09 · in From deep to long learning? -
…in a single GPU, which quite frankly seems like it should be impossible but somehow it works: https://github.com/glassroom/heinsen_routing#routing-very-lo... .
comment by cs702 · 2022-12-23 · in The Society of Mind (1986) [pdf]