Hacker GitHub

← All repos

BlinkDL/RWKV-LM

View on GitHub ↗ ★ 14,770 Python

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

33 mentions on Hacker News by 28 people: 3 stories, 30 comments · first 2022-05-11 · latest 2026-02-23

Mentions per year 33 mentions · busiest year 2023 (29)
20082026
AllStoriesComments NewestTopOldest
  1. …2025, peer-reviewed) Weights: https://huggingface.co/collections/RWKV/rwkv-v7-67d43835efa2... Code: https://github.com/BlinkDL/RWKV-LM (Apache 2.0) Happy to discuss the delta rule generalization, the TC⁰ proof, or the benchmark methodology…
    comment by Aedelon · 2026-02-23 · in RWKV-7 beats Llama 3.2 with 3x fewer training tokens and formally exceeds TC^0
  2. You can see all the development directly from them: https://github.com/BlinkDL/RWKV-LM Last week version 7 was released and every time they make significant improvements.
    comment by viraptor · 2025-02-14 · in Ask HN: Is anybody building an alternative transformer?
  3. …even with quantitization. people are simply trying to run from the transformer is seem. [1] https://github.com/BlinkDL/RWKV-LM
    comment by vicktorium · 2024-03-05 · in Based: Simple linear attention language models
  4. https://github.com/BlinkDL/RWKV-LM#rwkv-discord-httpsdiscord... lists a number of implementations of various versions of RWKV. https://github.com/BlinkDL/RWKV-LM#rwkv-parallelizable-rnn-w... : > RWKV: Parallelizable RNN with…
    comment by westurner · 2023-12-26 · in Ask HN: Do LLMs need a context window?
  5. …Models (v4 are the ones from the paper): https://huggingface.co/RWKV GitHub: https://github.com/BlinkDL/RWKV-LM
    comment by stellaathena · 2023-12-09 · in Paving the way to efficient architectures: StripedHyena-7B
  6. …forward to diving in! --- [a] https://en.wikipedia.org/wiki/State-space_representation#Lin... [b] https://github.com/BlinkDL/RWKV-LM [c] https://arxiv.org/abs/2006.16236
    comment by cs702 · 2023-12-05 · in Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  7. That is not completely true. There are RNNs with transformer/LLM-like performance. See e.g. https://github.com/BlinkDL/RWKV-LM . They are less popular, and less explored. But an interesting route ahead.
    comment by nextos · 2023-11-27 · in Understanding Deep Learning
  8. This is what RWKV ( https://github.com/BlinkDL/RWKV-LM ) was made for, and what it will be good at. Wow. Pretty darn cool! <3 :'))))
    comment by tysam_and · 2023-09-20 · in Q-Transformer: Scalable Reinforcement Learning via Autoregressive Q-Functions
  9. …all previous messages must be sent with each call. In this context, the RWKV-LM project on GitHub ( https://github.com/BlinkDL/RWKV-LM ) might offer a solution. They claim to achieve performance comparable to transformers using an RNN,…
    comment by machdiamonds · 2023-06-17 · in The Secret Sauce behind 100K context window in LLMs: all tricks in one place
  10. …got the training data, but they seem to have done everything mostly independently otherwise. https://github.com/BlinkDL/RWKV-LM disclaimer: I've been doing a lot of work lately on an implementation of CPU inference for this model, so I'm…
    comment by LoganDark · 2023-06-17 · in Meta's plan to offer free commercial AI models puts pressure on Google, OpenAI
  11. https://github.com/BlinkDL/RWKV-LM
    comment by int_19h · 2023-05-17 · in Numbers every LLM developer should know
  12. A better example would have been the RWKV [1] [1] https://github.com/BlinkDL/RWKV-LM
    comment by abidlabs · 2023-05-15 · in Brex’s Prompt Engineering Guide
  13. That's actually not true: https://github.com/BlinkDL/RWKV-LM
    comment by abidlabs · 2023-05-15 · in Brex’s Prompt Engineering Guide
  14. RWKV does: https://github.com/BlinkDL/RWKV-LM It uses „the Pile“: https://pile.eleuther.ai/ And I’ve seen some more in the last weeks.
    comment by kbrkbr · 2023-05-04 · in Google “We have no moat, and neither does OpenAI”
  15. Would be very interesting to see https://github.com/BlinkDL/RWKV-LM trained on the same data
    comment by logicchains · 2023-05-03 · in OpenLLaMA: An Open Reproduction of LLaMA
  16. …Others seem to agree: https://twitter.com/arankomatsuzaki/status/16390003799784038... . Code: https://github.com/BlinkDL/RWKV-LM * RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m…
    comment by cs702 · 2023-04-24 · in Scaling Transformer to 1M tokens and beyond with RMT
  17. …grants will be able to attest to this with multiple breakthroughs as a result, for example funding https://github.com/BlinkDL/RWKV-LM , the work of https://github.com/lucidrains and others. Similarly we funded the beta of MidJourney with a…
    comment by emadm · 2023-04-23 · in Leaked deck raises questions over Stability AI’s Series A pitch to investors
  18. …to have adapted a recursive model so it can be trained in a special parallel mode like attention. https://github.com/BlinkDL/RWKV-LM
    comment by _0ffh · 2023-04-18 · in The New XOR Problem
  19. I think https://github.com/BlinkDL/RWKV-LM could be used, but not all versions (namely instruction fine-tuned models trained on alpaca data)
    comment by K0IN · 2023-04-10 · in Ask HN: Open source LLM for commercial use?
  20. https://github.com/BlinkDL/RWKV-LM this claims to work well with long sequences.
    comment by logicchains · 2023-04-10 · in From deep to long learning?
  21. …compute resource such as RWKV[1], Open Assistant, some works by LAION[2] and lucidrains[3] [1] https://github.com/BlinkDL/RWKV-LM [2] https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K [3] https://github.com/lucidrains/gigagan-pyt…
    comment by blackcat201 · 2023-04-08 · in The company behind Stable Diffusion appears to be at risk of going under
  22. …[1] https://en.wikipedia.org/wiki/Transformer_(machine_learning_... [2] https://github.com/BlinkDL/RWKV-LM
    comment by wsgeorge · 2023-03-27 · in The Prospect of an AI Winter
  23. ▲ 328
    RWKV RNN: Better than ChatGPT?
    story by pffft8888 · 2023-03-23 · 107 comments
  24. …open assistant dataset, as this model is wicked fast in output and I'd love to see it fine tuned https://github.com/BlinkDL/RWKV-LM
    comment by amrb · 2023-03-10 · in George Hotz – Programming – can we fit a LLaMA inside a tinygrad?
  25. …RWKV [1] is the next step. It parallelizes even better and seems to have no sequence limit. [1] https://github.com/BlinkDL/RWKV-LM
    comment by naasking · 2023-03-01 · in Language Is Not All You Need: Aligning Perception with Language Models
  26. RWKV is showing that maybe RNNs can perform on par with transformers https://github.com/BlinkDL/RWKV-LM
    comment by macrolime · 2023-02-28 · in Language Is Not All You Need: Aligning Perception with Language Models
  27. …on, but does seem to have quite a lot more preprocessing to increase the dataset quality. [1] https://github.com/BlinkDL/RWKV-LM
    comment by nl · 2023-02-28 · in Open source implementation for LLaMA-based ChatGPT
  28. ▲ 5
    RWKV-LM: RNN with transformer-level performance, without using attention
    story by azhenley · 2023-02-27 · 0 comments
  29. …can get hands on, honestly this project is more interesting and lighting fast on even a 2060 laptop https://github.com/BlinkDL/RWKV-LM
    comment by amrb · 2023-02-27 · in Open source implementation for LLaMA-based ChatGPT
  30. https://github.com/BlinkDL/RWKV-LM
    comment by aPoCoMiLogin · 2023-02-14 · in Ask HN: Open-source ChatGPT alternatives?
  31. They explains it here: https://github.com/BlinkDL/RWKV-LM#the-rwkv-language-model-a...
    comment by eternalban · 2023-01-20 · in ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
  32. …really know either! But they reference https://arxiv.org/abs/2105.14103 and the bottom section of https://github.com/BlinkDL/RWKV-LM has an explainer.
    comment by nl · 2023-01-19 · in ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model
  33. ▲ 4
    RWKV-LM: RNN with transformer-level performance, without using attention
    story by vletal · 2022-05-11 · 0 comments