Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
Finding the sweet spot for training data without expensive trial and error
When training large language models, teams must split their annotation budget between two techniques: supervised fine-tuning (SFT) and reinforcement learning (RL). Rather than hunting for one perfect ratio, this work identifies a "near-optimal region"—a wide range of allocations that all perform nearly as well. The key finding: this region reliably transfers from small experimental models to large production models, so teams can run cheap small-scale experiments to figure out a good allocation without expensive large-model testing.
Training large language models is expensive, and knowing how to divide limited annotation resources between SFT and RL can save significant cost without sacrificing performance. This approach lets teams run quick experiments on small models to confidently set their budget split for large models, avoiding wasteful full-scale trial-and-error. The method also adapts when the cost difference between SFT and RL data changes, making it practical for real-world budget constraints.