tysam-code/hlb-gpt

View on GitHub ↗ ★ 361 Python

Minimalistic, extremely fast, and hackable researcher's toolbench for GPT models in 307 lines of code. Reaches <3.8 validation loss on wikitext-103 on a single A100 in <100 seconds. Scales to larger models with one parameter change (feature currently in alpha).

3 mentions on Hacker News by 1 person: 0 stories, 3 comments · first 2023-04-02 · latest 2023-05-23

Mentions per year 3 mentions · busiest year 2023 (3)

20082026

All Stories Comments Newest Top Oldest

…for a few reasons, generally seamless): https://github.com/tysam-code/hlb-CIFAR10 [for fp16] and https://github.com/tysam-code/hlb-gpt [for bf16] Personally from my experience, I think fp16/bf16 is honestly a bit too expressive for what we…
comment by tysam_and · 2023-05-23 · in In Defense of Pure 16-Bit Floating-Point Neural Networks
…converges to a decent loss in a short amount of time on an A100 -- just under 2.5 minutes or so: https://github.com/tysam-code/hlb-gpt With the original hyperparameters, it was 30-60 minutes, with a pruned down network and adjusted…
comment by tysam_and · 2023-04-06 · in Neural Networks: Zero to Hero
…(convolutional networks and imaging stuff. it's fast. very fast.) https://github.com/tysam-code/hlb-gpt (pruned-down base of nanoGPT with training-speed-focused changes built on top of it. Check out the 0.0.0 tag from the re
comment by tysam_and · 2023-04-02 · in MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention [video]