Category
Caution
Massive but extremely noisy; requires expensive filtering, dedupe, PII screening, and quality scoring.
Caution
Common Crawl
docs
unknown
Local AI / LLM Engineering / RAG
Local AI / LLM Engineering / RAG
Warning
Reference
publishable
avoid-or-link-only
no
no
nan
Warning
direct_links_master
direct_links_master, mega_open_hub
49c504710c17338b