← All questions
Training & Fine-tuningMedium· SFT· Context Window· Model Behavior

If a model ignores the context window and answers from...

This issue is categorized as a behavior failure rather than a knowledge failure. To fix it, the training data must be carefully curated. You should include examples where the golden document is present and used, as well as a deliberate fraction of examples where the document is removed. This teaches the model to avoid ‘hallucinating’ or relying on stale internal memory when relevant context is missing. A similar approach using counterfactual documents can address the mirror-image problem of the model swallowing confidently wrong information.

Code playground (Python)