Hacker GitHub

← All repos

kingoflolz/mesh-transformer-jax

View on GitHub ↗ ★ 6,389 Python

Model parallel transformers in JAX and Haiku

15 mentions on Hacker News by 14 people: 2 stories, 13 comments · first 2021-06-09 · latest 2023-03-22

Mentions per year 15 mentions · busiest year 2021 (6)
20082026
AllStoriesComments NewestTopOldest
  1. …tps://www.cstroik.com/index.php/2023/02/18/finetuning-an-a... - https://github.com/huggingface/peft - https://github.com/kingoflolz/mesh-transformer-jax/blob/mast... - https://github.com/oobabooga/text-generation-webui -…
    comment by lxe · 2023-03-22 · in Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
  2. …does training and serving with 175B parameter models https://github.com/alpa-projects/alpa - GPT-J https://github.com/kingoflolz/mesh-transformer-jax - Another HN thread on training LLMs with Ray (on TPUs in this case)…
    comment by robertnishihara · 2023-02-12 · in How to train large models on many GPUs? (2021)
  3. You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links The pretrained model is already available.
    comment by rockwotj · 2023-01-18 · in Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
  4. GPT-J specifically was written using Google Cloud's TPU stuff. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b They even have a google colab notebook thingie. Not much setup needed, just an account.…
    comment by mosseater · 2022-12-26 · in Roll out your own ChatGPT
  5. …of the smaller versions of bloom https://huggingface.co/docs/transformers/model_doc/bloom or gpt-j https://github.com/kingoflolz/mesh-transformer-jax
    comment by theCrowing · 2022-12-23 · in Ask HN: What language models can I fine-tune at home?
  6. They hinted at it in the screenshot, but the goods are linked from the https://6b.eleuther.ai page: https://github.com/kingoflolz/mesh-transformer-jax#gpt-j-6b (Apache 2)
    comment by mdaniel · 2022-12-15 · in GPT-J, an open-source alternative to GPT-3
  7. …and audio classification. --- Also I read about GPT-J, whose capability is comparable with GTP-3. https://github.com/kingoflolz/mesh-transformer-jax But I believe it requires buying or renting GPUs.
    comment by lioeters · 2022-12-12 · in Ask HN: Self-hosted/open-source ChatGPT alternative? Like Stable Diffusion
  8. doesnt seem the code is there - pretrained models are there. https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b https://huggingface.co/EleutherAI/gpt-j-6B isnt that so ?
    comment by sandGorgon · 2022-06-23 · in YaLM-100B: Pretrained language model with 100B parameters
  9. 175B * 16 bits = 350GB, but it does compress a bit. GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax , is 6B parameters but weighs 9GB. It does decompress to 12GB as expected. Assuming the same…
    comment by sanxiyn · 2022-05-03 · in OPT: Open Pre-trained Transformer Language Models
  10. I’m surprised that no one has answered for three hours! The answer is at https://github.com/kingoflolz/mesh-transformer-jax It has detailed comparisons and a full breakdown of the performance, courtesy of Eleuther.
    comment by sillysaurusx · 2021-11-07 · in GPT-3 is no longer the only game in town
  11. ▲ 1
    Fine-tuning GPT-J: key takeaways
    story by juliensalinas · 2021-09-23 · 0 comments
  12. …cloud. It’s a small version true but the results are almost on par with reported GPT-3 Source - https://github.com/kingoflolz/mesh-transformer-jax
    comment by sivakon · 2021-07-09 · in EleutherAI One Year Retrospective
  13. …before, but wouldn't one want to include a sha or git tag in such a citation? howpublished = {\url{https://github.com/kingoflolz/mesh-transformer-jax}}, I did see the month and year keywords, which I guess would narrow down the number of…
    comment by mdaniel · 2021-07-04 · in GPT-J-6B – A 6 billion parameter, autoregressive text generation model
  14. ▲ 636
    GPT-J-6B – A 6 billion parameter, autoregressive text generation model
    story by canada_dry · 2021-07-04 · 146 comments
  15. If you’re also curious how it does vs GPT3 here’s the Zero-shot performance evaluations table: https://github.com/kingoflolz/mesh-transformer-jax#zero-shot...
    comment by nefitty · 2021-06-09 · in GPT-J-6B: 6B JAX-Based Transformer