Hacker GitHub

← All repos

togethercomputer/RedPajama-Data

View on GitHub ↗ ★ 4,998 Python

The RedPajama-Data repository contains code for preparing large datasets for training large language models.

9 mentions on Hacker News by 9 people: 1 story, 8 comments · first 2023-04-17 · latest 2024-08-29

Mentions per year 9 mentions · busiest year 2023 (6)
20082026
AllStoriesComments NewestTopOldest
  1. …and a large dataset there's nothing stopping you. Download Red Pajama and Llama and give it a try. https://github.com/togethercomputer/RedPajama-Data
    comment by valine · 2024-08-29 · in Update on Llama adoption
  2. …100B text documents coming from 84 CommonCrawl snapshots and processed using the CCNet pipeline. https://github.com/togethercomputer/RedPajama-Data > Dolma Dataset: an open dataset of 3 trillion tokens from a diverse mix of web content,…
    comment by GeekyBear · 2024-04-26 · in Apple releases eight small AI language models aimed at on-device use
  3. …exhausting the text data we have. Here's a 5 language 30T token dataset from web scrapped data. https://github.com/togethercomputer/RedPajama-Data And that is just web scrapped data. There are trillions of valuable tokens worth of text…
    comment by og_kalu · 2024-04-03 · in Choose your weapon: Survival strategies for depressed AI academics
  4. …even any European power has the chance to win. [0] https://rom1504.github.io/clip-retrieval/ [1] https://github.com/togethercomputer/RedPajama-Data
    comment by godelski · 2023-12-05 · in AI and Mass Spying
  5. …from Hugging Face) - and many many others [1] https://mistral.ai/news/announcing-mistral-7b/ [2] https://github.com/togethercomputer/RedPajama-Data
    comment by ebalit · 2023-11-04 · in AI and Open Source in 2023
  6. …If you are interested in contributing the code for these features, feel free to do a PR to https://github.com/togethercomputer/RedPajama-Data ! Otherwise we will try our best effort implementation :) but we hope that this can become a…
    comment by zhangce · 2023-10-31 · in RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
  7. …on the tokenizer) "The Pile" has but couldn't find it. As far as I understand RedPajama has 1.2T ( https://github.com/togethercomputer/RedPajama-Data ) and has a table in the readme listing the main parts and how many tokens each part has.
    comment by tosh · 2023-06-10 · in The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  8. …following the original paper (allowing for some vagueness that Meta researchers didn't specify): https://github.com/togethercomputer/RedPajama-Data/ https://twitter.com/togethercompute/status/16479179892645191...
    comment by joshvm · 2023-04-30 · in A brief history of LLaMA models