← All questions
Training & Fine-tuningMedium· Data Curation· Instruction Tuning

How would you build an instruction dataset from scratch...

With limited resources, focus on quality and diversity rather than sheer scale. Start by repurposing existing labeled NLP datasets and applying templates to transform them into instruction-response pairs. Aim for a variety of task types, as 100 tasks with 10k examples each will generalize better than a single task with 1M examples. Once the model is stable, introduce synthetic or human-annotated open-ended conversational data to improve its flexibility.

Code playground (Python)