SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Testing AI coding assistants as they handle messy, changing requests from real users
Most coding agents are tested with single, polished requests—but real users change their minds and add requirements as they go. Researchers built SWE-DIVE, a testing system that simulates realistic multi-turn conversations between a user and a coding agent, complete with a "user simulator" that behaves like actual people do and a validator that checks whether the agent actually solved what was asked. The framework reveals how well agents handle the messiness of actual development work, not just textbook problems.
Coding assistants are moving into production where developers work iteratively, asking for changes and new features mid-session. Current benchmarks miss these real patterns entirely—an agent might pass a GitHub-issue test but fail when a user says "actually, make it faster" or "ignore what I said before." This framework closes that gap, letting companies test whether their tools will actually work for how people really code.