← All questions
LLM FoundationsMedium· Data Diversity· Domain Coverage· Pre-training

Why does The Pile use 22 sources instead of just filter...

Relying solely on Common Crawl, even with advanced filtering, is insufficient because it systematically underrepresents specific technical and academic domains. Sources like PubMed, arXiv, and Stack Exchange are not well-crawled by general web crawlers. Consequently, even the most sophisticated quality filters cannot extract information that is not present in the source data. While this advantage is currently eroding due to restricted API access and legal changes, using curated sources remains the best way to ensure domain-specific coverage.

Code playground (Python)