mistralai/megablocks-public
View on GitHub ↗ ★ 864 Python archived
4 mentions on Hacker News by 4 people: 1 story, 3 comments · first 2023-12-08 · latest 2023-12-08
20082026
-
…/ n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 Likely related code: https://github.com/mistralai/megablocks-public Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If…
comment by BryanLegend · 2023-12-08 · in Mistral "Mixtral" 8x7B 32k model [magnet] -
…"all demo no model" approach from earlier this week! Seems to be trained off Stanford's Megablocks: https://github.com/mistralai/megablocks-public
comment by cuuupid · 2023-12-08 · in Mistral "Mixtral" 8x7B 32k model [magnet] -
▲ 2
Mistral: Light-weight library for mixture-of-experts (MoE) trainingstory by georgehill · 2023-12-08 · 1 comment
-
Might be the training code related with the model https://github.com/mistralai/megablocks-public/tree/pstock/m...
comment by cloudhan · 2023-12-08 · in Mistral "Mixtral" 8x7B 32k model [magnet]