Hacker GitHub

← All repos

NVIDIA/Megatron-LM

View on GitHub ↗ ★ 18,349 Python

Ongoing research training transformer models at scale

8 mentions on Hacker News by 8 people: 0 stories, 8 comments · first 2019-08-13 · latest 2024-04-24

Mentions per year 8 mentions · busiest year 2024 (3)
20082026
AllStoriesComments NewestTopOldest
  1. https://github.com/NVIDIA/Megatron-LM This is probably a good baseline to start thinking about LLM training at scale.
    comment by spott · 2024-04-24 · in CoreNet: A library for training deep neural networks
  2. …SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training…
    comment by logicchains · 2024-04-19 · in Llama 3 8B is almost as good as Wizard 2 8x22B
  3. https://github.com/NVIDIA/Megatron-LM > Megatron is a large, powerful transformer [...]
    comment by johndough · 2024-02-23 · in Beyond A*: Better Planning with Transformers
  4. …scheduling? [1]: https://arxiv.org/abs/2006.09503 [2]: https://arxiv.org/abs/2101.06840 [3]: https://github.com/nvidia/Megatron-LM [4]: https://github.com/microsoft/DeepSpeed/issues/1110
    comment by sighingnow · 2023-10-10 · in Why async gradient update doesn't get popular in LLM community?
  5. GPU cluster scaling has come a long way. Just check out the scaling plot here: https://github.com/NVIDIA/Megatron-LM
    comment by rerx · 2023-04-27 · in Why did Google Brain exist?
  6. …the low hanging fruit of silicon that has provided such a bounteous harvest for the last 70 years. https://github.com/NVIDIA/Megatron-LM
    comment by ArtWomb · 2021-04-05 · in Semiconductor Startups – Are they back?
  7. …(basically a locality-sensitive hash to detect near-duplicates). * I used the [Megatron-LM]( https://github.com/NVIDIA/Megatron-LM ) code for training. Training took about 1 month on 4x RTX8000 GPUs. * You can download the trained model…
    comment by moyix · 2021-02-23 · in Try to guess if code is real or GPT2-generated
  8. https://github.com/NVIDIA/Megatron-LM
    comment by thats_legit · 2019-08-13 · in Megatron-LM: Nvidia's 8.3B Parameter GPT-2 Transformer