RedPajama

Category

Caution

Provider

Hugging Face

Type

dataset

Level

unknown

Topic

Local AI / LLM Engineering / RAG

Track

Local AI / LLM Engineering / RAG

Section

Warning

Format

Reference

Status

publishable

Commercial

avoid-or-link-only

Featured

no

Fast start

no

Sequence

nan

Priority

Warning

Primary source

direct_links_master

Sources

direct_links_master, mega_open_hub, training_data_stack

ID

6bf6e3b4f4ec0bf7

OpenWebText

Hugging Face

Curators note they do not own the underlying text and maintain a takedown process.

Hugging Face Datasets

Hugging Face

Massive public dataset hub spanning NLP, code, vision, audio, robotics, and benchmarks.

FineWeb2

Hugging Face

Best multilingual extension of FineWeb pipeline; very broad language coverage.

FineWeb

Hugging Face

Huge cleaned English web corpus; best raw breadth for LLM pretraining.

Cosmopedia

Hugging Face

Synthetic textbook/blog/WikiHow-style corpus that helps tutor-like explanations.

Common Pile v0.1

Hugging Face

Best legally cleaner starting corpus: 8 TB of public-domain and openly licensed text spanning books, papers, code, encyclopedias, educational materials, and transcripts.

Hugging Face Datasets documentation

Hugging Face

Essential for loading, cleaning, streaming, and publishing training / eval data.

Stanford AIMI Shared Datasets

aimi.stanford.edu

Excellent medical data, but shared data is broadly non-commercial.

Semantic Scholar full datasets

api.semanticscholar.org

Public dataset license is limited to internal, non-commercial research/education use.

xP3

Hugging Face

Crosslingual prompt pool across many languages and tasks.

Vision-Flan

Hugging Face

Good open visual instruction tuning layer after base vision-language pretraining.

UltraChat

Hugging Face

Good synthetic multi-turn chat augmentation.