togethercomputer/RedPajama-Data
View on GitHub ↗ ★ 4,998 Python
The RedPajama-Data repository contains code for preparing large datasets for training large language models.
9 mentions on Hacker News by 9 people: 1 story, 8 comments · first 2023-04-17 · latest 2024-08-29
20082026
-
…and a large dataset there's nothing stopping you. Download Red Pajama and Llama and give it a try. https://github.com/togethercomputer/RedPajama-Data
comment by valine · 2024-08-29 · in Update on Llama adoption -
…100B text documents coming from 84 CommonCrawl snapshots and processed using the CCNet pipeline. https://github.com/togethercomputer/RedPajama-Data > Dolma Dataset: an open dataset of 3 trillion tokens from a diverse mix of web content,…
comment by GeekyBear · 2024-04-26 · in Apple releases eight small AI language models aimed at on-device use -
…exhausting the text data we have. Here's a 5 language 30T token dataset from web scrapped data. https://github.com/togethercomputer/RedPajama-Data And that is just web scrapped data. There are trillions of valuable tokens worth of text…
comment by og_kalu · 2024-04-03 · in Choose your weapon: Survival strategies for depressed AI academics -
…even any European power has the chance to win. [0] https://rom1504.github.io/clip-retrieval/ [1] https://github.com/togethercomputer/RedPajama-Data
comment by godelski · 2023-12-05 · in AI and Mass Spying -
…from Hugging Face) - and many many others [1] https://mistral.ai/news/announcing-mistral-7b/ [2] https://github.com/togethercomputer/RedPajama-Data
comment by ebalit · 2023-11-04 · in AI and Open Source in 2023 -
…If you are interested in contributing the code for these features, feel free to do a PR to https://github.com/togethercomputer/RedPajama-Data ! Otherwise we will try our best effort implementation :) but we hope that this can become a…
comment by zhangce · 2023-10-31 · in RedPajama v2 Open Dataset with 30T Tokens for Training LLMs -
…on the tokenizer) "The Pile" has but couldn't find it. As far as I understand RedPajama has 1.2T ( https://github.com/togethercomputer/RedPajama-Data ) and has a table in the readme listing the main parts and how many tokens each part has.
comment by tosh · 2023-06-10 · in The Pile: An 800GB Dataset of Diverse Text for Language Modeling -
…following the original paper (allowing for some vagueness that Meta researchers didn't specify): https://github.com/togethercomputer/RedPajama-Data/ https://twitter.com/togethercompute/status/16479179892645191...
comment by joshvm · 2023-04-30 · in A brief history of LLaMA models -
▲ 2
RedPajama-Data: Code for preparing large datasetsstory by harrisonpowers · 2023-04-17 · 0 comments