Hacker GitHub

← All repos

jaymody/picoGPT

View on GitHub ↗ ★ 3,483 Python

An unnecessarily tiny implementation of GPT-2 in NumPy.

5 mentions on Hacker News by 4 people: 1 story, 4 comments · first 2023-02-23 · latest 2025-02-05

Mentions per year 5 mentions · busiest year 2023 (4)
20082026
AllStoriesComments NewestTopOldest
  1. I'm curious, what does paradigmatic JAX look like? Is there an equivalent of picoGPT [1] for JAX? [1] https://github.com/jaymody/picoGPT/blob/main/gpt2.py
    comment by Scene_Cast2 · 2025-02-05 · in How to scale your model: A systems view of LLMs on TPUs
  2. In that case, you might also enjoy https://jaykmody.com/blog/gpt-from-scratch/ (here's the raw code: https://github.com/jaymody/picoGPT/blob/main/gpt2.py )
    comment by Scene_Cast2 · 2023-08-25 · in Understanding Automatic Differentiation in 30 lines of Python
  3. I wrote a minimal implementation in NumPy here (the forward pass code is only 40 lines): https://github.com/jaymody/picoGPT And also a related blog post: https://news.ycombinator.com/item?id=34726115 Although this is for a decoder-only…
    comment by jaymody · 2023-04-26 · in Transformers from Scratch (2021)
  4. Take a look at https://github.com/jaymody/picoGPT/blob/a750c145ba4d09d57648... Yes, this is GPT-2 not 4 and it‘s not the RL-trained Chat, only the GPT model and it‘s basically only the inference part, not the training loop and it‘s…
    comment by quonn · 2023-03-31 · in Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality