OpenWebText

Category

Caution

Provider

Hugging Face

Type

dataset

Level

unknown

Topic

Local AI / LLM Engineering / RAG

Track

Local AI / LLM Engineering / RAG

Section

Warning

Format

Reference

Status

publishable

Commercial

avoid-or-link-only

Featured

no

Fast start

no

Sequence

nan

Priority

Warning

Primary source

direct_links_master

Sources

direct_links_master, mega_open_hub

ID

ab8f5de35d181327

RedPajama

Hugging Face

Mixed-source replication corpus including CommonCrawl, C4, GitHub, arXiv, Wikipedia, and StackExchange.

Hugging Face Datasets

Hugging Face

Massive public dataset hub spanning NLP, code, vision, audio, robotics, and benchmarks.

FineWeb2

Hugging Face

Best multilingual extension of FineWeb pipeline; very broad language coverage.

FineWeb

Hugging Face

Huge cleaned English web corpus; best raw breadth for LLM pretraining.

Cosmopedia

Hugging Face

Synthetic textbook/blog/WikiHow-style corpus that helps tutor-like explanations.

Common Pile v0.1

Hugging Face

Best legally cleaner starting corpus: 8 TB of public-domain and openly licensed text spanning books, papers, code, encyclopedias, educational materials, and transcripts.

Hugging Face Datasets documentation

Hugging Face

Essential for loading, cleaning, streaming, and publishing training / eval data.

Stanford AIMI Shared Datasets

aimi.stanford.edu

Excellent medical data, but shared data is broadly non-commercial.

Semantic Scholar full datasets

api.semanticscholar.org

Public dataset license is limited to internal, non-commercial research/education use.

xP3

Hugging Face

Crosslingual prompt pool across many languages and tasks.

Vision-Flan

Hugging Face

Good open visual instruction tuning layer after base vision-language pretraining.

UltraChat

Hugging Face

Good synthetic multi-turn chat augmentation.