← All questions
Inference & ServingMedium· Inference Optimization· Reasoning Models

How would you control the inference cost of a reasoning...

To control inference costs, introduce a length reward during the RL phase that incentivizes correct-and-short responses over verbose ones, ideally after initial convergence to avoid local minima. For serving, implement thinking mode fusion, which allows operators to set a hard cap on the thinking-token budget at inference time, ensuring graceful degradation rather than abrupt failure.

Code playground (Python)