facebookresearch/metaseq on Hacker News

https://github.com/facebookresearch/metaseq/blob/main/projec... I really enjoyed reading the log book from the training of OPT-175B at Meta… I guess it’s all classified info but it’d be fun to read a blog post about the crazy day to day…

comment by costco · 2026-10-05 · in Beam: Reflection's 501B open-weight model

…releases a more technical report about their training adventures, like the PDF file here: https://github.com/facebookresearch/metaseq/tree/main/projec...

comment by pama · 2024-03-06 · in Training LLMs from ground zero as a startup

▲ 1

Chronicles of Opt Development

story by tosh · 2024-03-06 · 0 comments

Nevermind, I figured it out: https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by nalzok · 2023-11-15 · in Model training diary/journal for LLMs?

…resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by patapong · 2023-10-19 · in Consortium launched to build the largest open LLM

…makes it very difficult to re-implement in another language (e.g., Julia) or framework. 0: Such as https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by techwizrd · 2023-07-20 · in The open source learning curve for AI researchers

…a major feat with many tweaks and tricks. Case in point: the 114 pages of OPT175B logbook [1] [1] https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by whiplash451 · 2023-06-22 · in See the pitch memo that raised €105M for four-week-old startup Mistral

…all indications are that LLaMA 2 would be fully FLOSS. If you meant OPT, it isn't libre either: https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by ignoramous · 2023-06-16 · in Meta's plan to offer free commercial AI models puts pressure on Google, OpenAI

…meanwhile LLMs comparable to GPT-3/ChatGPT have had pretty insane memory requirements - e.g., < https://github.com/facebookresearch/metaseq/issues/146 >

comment by OmarAssadi · 2023-03-10 · in Visual ChatGPT

It seems so, check this log book from Meta: https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by RicoElectrico · 2023-02-20 · in Ask HN: Is There On-Call in ML?

▲ 1

Meta OPT-175B LLM training logbook [pdf]

story by HillRat · 2023-02-10 · 0 comments

…hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec ... Sounds like a good opportunity to learn. No pain, no gain :-) > 4. follow the…

comment by vvrm · 2023-01-20 · in Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

…hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec... 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the…

comment by karpathy · 2023-01-18 · in Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]

…as well. Here is an example of the engineering capabilities required to train the modern models: https://github.com/facebookresearch/metaseq/blob/main/projec... [PDF] It's a 114 page document detailing the months of full-time work for…

comment by oliwary · 2023-01-12 · in Ask HN: Can you crowdfund the compute for GPT?

…schedulers, it's a whole bunch of stuff. See this logbook from training the GPT-3 sized OPT model - https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by andreyk · 2022-12-11 · in The GPT Architecture, on a Napkin

…). There’s a logbook associated with the training of OPT (similar size to GPT-3): https://github.com/facebookresearch/metaseq/blob/main/projec... This article estimated the cost of training GPT3 to be over $4M: https://heits.digital/article

comment by brrrrrm · 2022-12-07 · in Ask HN: Training my own chat gpt

…anyone with a .edu address (models up to 66B are freely available) and Bloom-176B is 100% open: https://github.com/facebookresearch/metaseq https://huggingface.co/bigscience/bloom

comment by moyix · 2022-09-22 · in Whisper – open source speech recognition by OpenAI

…[1] https://www.youtube.com/watch?v=YQ2QtKcK2dA [2] https://github.com/yandex/YaLM-100B [3] https://github.com/facebookresearch/metaseq

comment by benreesman · 2022-08-21 · in Beyond Hyperanthropomorphism

…had also released a (much more detailed) "Logbook" when they published their 175b models in May: https://github.com/facebookresearch/metaseq/blob/main/projec... For example I somehow expected a Chinese research group with such a proven…

comment by MasterScrat · 2022-08-05 · in GLM-130B: An Open Bilingual (English and Chinese) Pre-Trained Language Model

…detailing their training process for a model that is similar in size. Does anyone have a link? Is https://github.com/facebookresearch/metaseq/blob/main/projec... what you're referring to?

comment by ninjaranter · 2022-07-20 · in DALL·E now available in beta

▲ 3

Meta uses Hacker News (and more) in OPT (its version of GPT-3)

story by curiousmindz · 2022-06-28 · 0 comments

The logbook is awesome: https://github.com/facebookresearch/metaseq/blob/main/projec... This is the true secret sauce -- all the tricks on how to get these things to train properly that aren't really published.

comment by nharada · 2022-06-27 · in Meta is inviting researchers to pick apart the flaws in its version of GPT-3

…on any platform - for example, the logbooks that Meta recently published are fascinating: https://github.com/facebookresearch/metaseq/blob/main/projec... We aim for Cloud TPUs to simplify the process of training models at these scales and…

comment by zak · 2022-05-11 · in Cloud TPU VMs are generally available

▲ 1

Metaseq: Repo for external large-scale work

story by pestkranker · 2022-05-05 · 0 comments

▲ 1

Chronicles of Opt-175B Training

story by echen · 2022-05-04 · 0 comments

…rendering [0] of the communal log book that facebook researchers kept while training opt-175. [0] https://github.com/facebookresearch/metaseq/tree/main/projec...

comment by Nzen · 2022-05-04 · in 100 Pages of raw notes released with the language model OPT-175

▲ 4

Facebook Research's logbook for training a 175B parameter language model [pdf]

story by yeldarb · 2022-05-04 · 0 comments

▲ 1

Chronicles of Opt-175B Training

story by maxrobeyns · 2022-05-04 · 0 comments

▲ 1

Training OPT-175 Meta raw log book [pdf]

story by deep_playa · 2022-05-03 · 0 comments

And it’s live! https://github.com/facebookresearch/metaseq Logbook links in specific: https://github.com/facebookresearch/metaseq/blob/main/projec...

comment by moyapchen · 2022-05-03 · in OPT: Open Pre-trained Transformer Language Models

…will be open "First thing in AM" ( https://twitter.com/stephenroller/status/1521302841276645376 ): https://github.com/facebookresearch/metaseq/

comment by MasterScrat · 2022-05-03 · in OPT: Open Pre-trained Transformer Language Models