Hacker GitHub

← All repos

KellerJordan/modded-nanogpt

View on GitHub ↗ ★ 6,002 Python

NanoGPT (124M) in 90 seconds

21 mentions on Hacker News by 16 people: 4 stories, 17 comments · first 2024-10-12 · latest 2026-09-29

Mentions per year 21 mentions · busiest year 2025 (13)
20082026
AllStoriesComments NewestTopOldest
  1. …was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction. 1. https://github.com/KellerJordan/modded-nanogpt#world-record-...
    comment by kelseyfrog · 2026-08-23 · in NanoGPT Speedrun Frontier
  2. …loses some rigor on the way) [2] 1 https://github.com/karpathy/nanochat/blob/master/dev/LOG.md 2 https://github.com/kellerjordan/modded-nanogpt
    comment by wongarsu · 2026-04-05 · in Nanocode: The best Claude Code that $200 can buy in pure JAX on TPUs
  3. Cool post. And you might find it interesting. https://github.com/KellerJordan/modded-nanogpt
    comment by recrush · 2026-01-28 · in Training a 67M-parameter transformer on an M4 Mac Mini
  4. …speedrun. It's much easier to train, and gives you a better comparison vs optimized systems. https://github.com/KellerJordan/modded-nanogpt
    comment by Herring · 2026-01-18 · in Starting from scratch: Training a 30M Topological Transformer
  5. …without changing the activation function, and then this project was based on nanochat. [1] -- https://github.com/KellerJordan/modded-nanogpt
    comment by kouteiheika · 2025-11-15 · in Show HN: Tiny Diffusion – A character-level text diffusion model from scratch
  6. …reasoning problems here, provided that they’re used in the context of a coding agent tool. [1] https://github.com/KellerJordan/modded-nanogpt
    comment by alyxya · 2025-10-31 · in Reasoning models reason well, until they don't
  7. …transformer. That is fine! But at least it will make the comparison actually useful . [1] -- https://github.com/KellerJordan/modded-nanogpt [2] -- https://kellerjordan.github.io/posts/muon/#discussion-solvin...
    comment by kouteiheika · 2025-10-22 · in The Dragon Hatchling: The missing link between the transformer and brain models
  8. …Notably it uses the author's Muon optimizer [2], rather than AdamW, (for the linear layers). [1] https://github.com/KellerJordan/modded-nanogpt [2] https://kellerjordan.github.io/posts/muon/
    comment by montebicyclelo · 2025-10-13 · in NanoChat – The best ChatGPT that $100 can buy
  9. …), setting a modded-nanogpt speed record and helping with several others (5.03m -> 4.66m, https://github.com/KellerJordan/modded-nanogpt/blob/master/r ... ), improving the relative loss of DiLoCo speedrunning on modded-nanogpt by over 40%…
    comment by fern_ · 2025-09-06 · in Ask HN: Who wants to be hired? (September 2025)
  10. …models of the same size trained on the same data. For example, small-model speedruns are a thing: https://github.com/KellerJordan/modded-nanogpt
    comment by yorwba · 2025-08-30 · in From multi-head to latent attention: The evolution of attention mechanisms
  11. https://github.com/KellerJordan/modded-nanogpt is pretty great in that respect
    comment by pizza · 2025-08-14 · in What's the strongest AI model you can train on a laptop in five minutes?
  12. …GPT-2 speedrun effort [0], at minimum Muon, better init and carefully tuning learning rate. [0]: https://github.com/KellerJordan/modded-nanogpt
    comment by tootyskooty · 2025-08-14 · in What's the strongest AI model you can train on a laptop in five minutes?
  13. …//x.com/karpathy/status/1620103412686942208 ), setting a modded-nanogpt speed record (5.03m -> 4.66m, https://github.com/KellerJordan/modded-nanogpt/blob/master/r... ), improving the relative loss of DiLoCo speedrunning on modded-nanogpt…
    comment by fern_ · 2025-08-01 · in Ask HN: Who wants to be hired? (August 2025)
  14. …the rules but still very interesting IMHO - definitely something I'd like to read paper about) https://github.com/KellerJordan/modded-nanogpt Also, I read yesterday that at some point, the embeddings across all of the models are (very)…
    comment by cztomsik · 2025-05-22 · in Gemini Diffusion
  15. It's a fully custom architecture, heavily inspired by the modded-nanogpt speedrun ( https://github.com/KellerJordan/modded-nanogpt ) but written fully from scratch and further tweaked/modified. I use it for experiments and as a testbed…
    comment by kouteiheika · 2025-03-15 · in Transformers Without Normalization
  16. …architecture (and broader software) changes. Here's a fun project I've had my eye on recently: https://github.com/KellerJordan/modded-nanogpt/
    comment by Herring · 2025-01-22 · in OpenAI has upped its lobbying efforts nearly sevenfold
  17. …rented 8xH100 80GB node (~$9/hr on Lambda Labs, RunPod io, etc.) using the NanoGPT speed run repo: https://github.com/KellerJordan/modded-nanogpt For that short of a run, you'll spend more time waiting for the node to come up, downloading…
    comment by smaddox · 2025-01-21 · in Ask HN: Is anyone doing anything cool with tiny language models?