← All questions
Data, Privacy & LegalHard· Data Augmentation· Data Quality

Your quality filter throws away 98% of your Common Craw...

To increase usable data without additional crawling, you should employ rephrasing and task conversion. Rephrasing involves rewriting low-quality documents to improve their utility, while task conversion involves transforming raw text into specific instruction-following formats. It is important to distinguish between these two methods and quantify the expected yield increase against the compute cost of the transformation process.

Code playground (Python)