Hacker GitHub

← All repos

karpathy/nanoGPT

View on GitHub ↗ ★ 64,364 Python

The simplest, fastest repository for training/finetuning medium-sized GPTs.

49 mentions on Hacker News by 48 people: 9 stories, 40 comments · first 2023-01-03 · latest 2026-09-30

Mentions per year 49 mentions · busiest year 2023 (25)
20082026
AllStoriesComments NewestTopOldest
  1. …if you want to scale the experiments up at home you need to go faster. Karpathy made optimized https://github.com/karpathy/nanoGPT , but it is tuned for "8XA100 40GB node in about 4 days of training". 13s is a bit overkill here (my machine…
    comment by lostmsu · 2026-09-30 · in Show HN: TurboGPT: train 22KiB transformer in 13s
  2. This weekend I just cracked into nanoGPT ( https://github.com/karpathy/nanoGPT ), an older but fabulous learning exercise where you build and train a crappy shakespeare GPT with ~0.8M parameters on a cpu. Results are about what you'd…
    comment by faxmeyourcode · 2025-10-13 · in NanoChat – The best ChatGPT that $100 can buy
  3. Andrej Karpathy's Nano GPT is reasonably accessible and easy to run. https://github.com/karpathy/nanoGPT
    comment by beardyw · 2025-10-10 · in Ask HN: Build Your Own LLM?
  4. …is probably the most rudimentary, only converting the characters of the input into integers. e.g. https://github.com/karpathy/nanoGPT/blob/master/data/shakesp...
    comment by z3c0 · 2025-05-30 · in Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
  5. …artifact of a bygone era. Try HuggingFace https://github.com/huggingface/course or Karpathy's intro https://github.com/karpathy/nanoGPT
    comment by FilosofumRex · 2025-03-19 · in Learning about Innovation from Half a Century of Conway's Game of Life
  6. …on tinystories in stead of Shakespeare. smolGPT also looks inspired by nanoGPT, also from Karpathy: https://github.com/karpathy/nanoGPT/blob/master/train.py
    comment by zerd · 2025-01-30 · in SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch
  7. …isn't true. Learning how to train a 124M is just as useful as a 700B, and is possible on a laptop. https://github.com/karpathy/nanoGPT
    comment by fzzzy · 2025-01-30 · in Mistral Small 3
  8. …training pipeline using Shakespeare. [1] The use case for this is learning in simple example. [1] https://github.com/karpathy/nanoGPT
    comment by efm · 2025-01-30 · in SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch
  9. github has a bunch of them for years, the most known from Andrej Karpathy: https://github.com/karpathy/nanoGPT some other have MoE implemented.
    comment by numba888 · 2025-01-29 · in SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch
  10. …try training your own GPT to have a sense for how it works. I wrote a Rust version of the famous https://github.com/karpathy/nanoGPT (which is in Python) so that I could learn how it's built. I wrote it in Rust because I wanted to improve…
    comment by airstrike · 2025-01-24 · in AI isn't going to kill the software industry
  11. …vanilla gpt from scratch with the same total embeddings size as your control. I would start with https://github.com/karpathy/nanoGPT as a baseline since you can train a toy (GPT2 sized) llm in a couple days on an a100 which are pretty easy…
    comment by mattnewton · 2024-10-24 · in Probably pay attention to tokenizers
  12. …I wanted to succinctly say "practically, quite a bit". Here's a good little rabbit-hole example: > https://github.com/karpathy/nanoGPT/blob/master/model.py#L45 Karpathy's nanoGPT calling flash attention by checking if…
    comment by vhiremath4 · 2024-07-11 · in FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision
  13. …networks (aka "multi-layer perceptron") and to more-sophisticated models like transformers (e.g. https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436... ). It's less about combining lots of tiny local linear transformations…
    comment by nerdponx · 2024-05-14 · in GPT-4o
  14. Where is the GPT implementation in JAX? I only found this [1] in PyTorch and NumPy. [1] https://github.com/karpathy/nanoGPT
    comment by auraham · 2024-04-09 · in Llm.c – LLM training in simple, pure C/CUDA
  15. …Seeing this has inspired me further to consider working on that project again at some point. [0] https://github.com/karpathy/nanoGPT [1] https://en.wikipedia.org/wiki/Trie [2] https://en.wikipedia.org/wiki/N-Triples [3]…
    comment by throwaway4aday · 2024-03-01 · in Show HN: Predictive text using only 13kb of JavaScript. no LLM
  16. Not sure what you mean with SLM, but https://github.com/karpathy/nanoGPT
    comment by sva_ · 2024-02-16 · in Building an LLM from Scratch: Automatic Differentiation (2023)
  17. The most recent is this, which I believe was made after he left Tesla: https://github.com/karpathy/nanoGPT And it's accompanying video series: https://karpathy.ai/zero-to-hero.html Another example (although I honestly don't remember if he…
    comment by Imnimo · 2024-02-14 · in Hi everyone yes, I left OpenAI yesterday
  18. …there's been in LLMs, has great simple tutorials right there eg. fine-tuning GPT2 with Shakespeare: https://github.com/karpathy/nanoGPT
    comment by speps · 2024-01-05 · in A simulation of me: fine-tuning an LLM on 240k text messages
  19. For training from scratch, maybe a small model like https://github.com/karpathy/nanoGPT or tinyllama. Perhaps with quantization. Fine-tuning is very doable. The hard part is making a novel dataset with input output pairs. You might…
    comment by ilaksh · 2024-01-03 · in Ask HN: Is it feasible to train my own LLM?
  20. …project[2] is handy if you want to dip your toe into training. 1. https://simonwillison.net/ 2. https://github.com/karpathy/nanoGPT/
    comment by runjake · 2024-01-02 · in Ask HN: Is it feasible to train my own LLM?
  21. …for what it's recreating. (I don't know how fair the comparison to other loc figures mentioned is). https://github.com/karpathy/nanoGPT
    comment by andy99 · 2023-12-11 · in GigaGPT: GPT-3 sized models in 565 lines of code
  22. …and training code in ~600 lines (although networks have easier logic to code than a compiler). [1] https://github.com/karpathy/nanoGPT
    comment by sciolist · 2023-09-04 · in Writing a C compiler in 500 lines of Python
  23. …or in the final linear layer before the softmax. You can see how it’s implemented in nanoGPT: https://github.com/karpathy/nanoGPT/blob/f08abb45bd2285627d1... At training time, probabilities for the next token are computed for each…
    comment by rlanday · 2023-07-25 · in Attention Is Off By One
  24. It would be interesting to see: (1) why a lamini account is required? (2) how this compares to https://github.com/karpathy/nanoGPT
    comment by adr1an · 2023-07-12 · in Free and Fast LLM Finetuning
  25. …system does not contradict his statement. Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L12... The model is only trained to predict the next token . The training regime…
    comment by gamegoblin · 2023-07-03 · in Douglas Hofstadter changes his mind on Deep Learning and AI risk
  26. Thanks, Is there a toy conversational LLM on Github or elsewhere? Something like: https://github.com/karpathy/nanoGPT
    comment by JPLeRouzic · 2023-06-13 · in Ask HN: How does an LLM “know” to respond in the first place?
  27. ….com/TikkunCreation/5de1df7b24800cc05b482... In particular, you'll probably want to skip to nanoGPT ( https://github.com/karpathy/nanoGPT ) and then maybe if you are interested in a bit more of the theory, Zero to Hero (…
    comment by tikkun · 2023-06-02 · in Ask HN: Resource to learn how to train and use ML Models
  28. …ithub.com/karpathy/randomfun/blob/master/lectures/m... [4]: https://github.com/karpathy/makemore [5]: https://github.com/karpathy/nanoGPT
    comment by sirodoht · 2023-04-05 · in Neural Networks: Zero to Hero
  29. Correct, the original is GPL 3. To produce this implementation from the LLaMA paper we started from github.com/karpathy/nanoGPT, the LLaMA architecture is really similar to GPT. For instance we added rotary positional encoding starting…
    comment by lantiga · 2023-03-28 · in Show HN: A fully open-source (Apache 2.0)implementation of llama
  30. Just playing casually with NanoGPT ( https://github.com/karpathy/nanoGPT ) with a desktop holding a 2080ti, it's really really really clear to me that the path to get to a pre-fine-tuned LLM is remarkably easy. RLHF is the piece above this…
    comment by aliljet · 2023-03-26 · in Using ChatGPT Plugins with LLaMA
  31. …won't fit in memory. If you could get it to run, it would be extremely slow Worth checking out: https://github.com/karpathy/nanoGPT The associated videos will go into this more, iirc, at the end he said the real GPT-2 would take 5-20…
    comment by verdverm · 2023-02-16 · in How fast would ChatGPT run on a laptop locally?
  32. …his own minimal GPT ( https://github.com/karpathy/minGPT ), and the much smaller version of it ( https://github.com/karpathy/nanoGPT ). But of course, since we are in this blog post I would refer you to this 60 line numpy GPT first (A. to…
    comment by tysam_and · 2023-02-09 · in A GPT in 60 Lines of NumPy
  33. this is pretty small: https://github.com/karpathy/nanoGPT
    comment by sharemywin · 2023-02-09 · in A GPT in 60 Lines of NumPy
  34. Can you please run bench from https://github.com/karpathy/nanoGPT ?
    comment by lostmsu · 2023-01-18 · in Which GPU(s) to get for deep learning
  35. …- Many smart developers try to reduce code, not multiply it: https://github.com/geohot/tinygrad https://github.com/karpathy/nanoGPT - Satisficing https://en.wikipedia.org/wiki/Satisficing - Perfect is the enemy of good…
    comment by hosolmaz · 2023-01-12 · in Ask HN: Strategies for working with engineers that are too smart?
  36. …size matters a lot more than parameter count, and nanoGPT author is trying to reproduce that here: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...
    comment by naasking · 2023-01-11 · in NanoGPT
  37. https://github.com/karpathy/nanoGPT#i-only-have-a-macbook > This creates a much smaller Transformer (4 layers, 4 heads, 64 embedding size), runs only on CPU, does not torch.compile the model (torch seems to give an error if you try), only…
    comment by JustSomeNobody · 2023-01-11 · in NanoGPT
  38. …In a real system, what you want to optimize for is accuracy, composability, inventiveness. [0]: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...
    comment by espadrine · 2023-01-11 · in NanoGPT
  39. I think the link should be: https://github.com/karpathy/nanoGPT
    comment by iamflimflam1 · 2023-01-11 · in NanoGPT
  40. what about https://github.com/karpathy/nanoGPT (currently trending on Github)
    comment by ladino · 2023-01-10 · in Microsoft eyes $10B bet on ChatGPT