Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Making AI feedback six times more efficient for image generation models
Training image generation models to match human preferences currently wastes feedback by treating all learning moments equally. This paper shows that some timesteps in the generation process carry much more useful information than others, and some past examples are more worth revisiting — together, these insights reduce the amount of human feedback needed by up to sixfold while maintaining quality.
Human feedback is expensive and slow to collect. By slashing feedback requirements by up to 6×, this approach makes it practical to fine-tune image generators in real-world settings where human judgments are the limiting resource. This could accelerate the development of personalized AI models that better match what individual users or organizations actually want.