Semantic Scholar full datasets

Category

Caution

Provider

api.semanticscholar.org

Type

dataset

Level

unknown

Topic

Local AI / LLM Engineering / RAG

Track

Local AI / LLM Engineering / RAG

Section

Warning

Format

Reference

Status

manual_review

Commercial

avoid-or-link-only

Featured

no

Fast start

no

Sequence

nan

Priority

Warning

Primary source

direct_links_master

Sources

direct_links_master, mega_open_hub

ID

1174d0593cc678d0

Stanford AIMI Shared Datasets

aimi.stanford.edu

Excellent medical data, but shared data is broadly non-commercial.

Hugging Face Datasets

Hugging Face

Massive public dataset hub spanning NLP, code, vision, audio, robotics, and benchmarks.

Hugging Face Datasets documentation

Hugging Face

Essential for loading, cleaning, streaming, and publishing training / eval data.

UCI Datasets

UCI

Direct browser for UCI datasets when you want a clean, filterable dataset list for website linking.

RedPajama

Hugging Face

Mixed-source replication corpus including CommonCrawl, C4, GitHub, arXiv, Wikipedia, and StackExchange.

OpenWebText

Hugging Face

Curators note they do not own the underlying text and maintain a takedown process.

MIT OpenCourseWare

MIT

Excellent lecture notes, exams, and videos across advanced technical topics.

Common Pile v0.1

Hugging Face

Best legally cleaner starting corpus: 8 TB of public-domain and openly licensed text spanning books, papers, code, encyclopedias, educational materials, and transcripts.

UCI Machine Learning Repository

UCI

Classic and modern ML datasets that are ideal for education, benchmarking, and tabular experiments.

OpenML Docs

OpenML

Open ecosystem for datasets, tasks, models, and runs that helps standardize ML benchmarking.

FineWeb2

Hugging Face

Best multilingual extension of FineWeb pipeline; very broad language coverage.

FineWeb

Hugging Face

Huge cleaned English web corpus; best raw breadth for LLM pretraining.