FDE-Gym: The Last Mile of AI Delivery
Why AI agents struggle when customers don't know what they actually need
AI agents that excel at well-defined lab tasks fail dramatically in real customer work, solving only 14 percent of realistic delivery problems. The main culprit is uncovering hidden constraints — what customers didn't mention because they didn't know to mention it — but even when agents gather all the facts, most still can't turn that knowledge into working solutions.
As companies deploy AI to solve actual business problems, this gap between benchmark performance and real-world delivery directly impacts whether customers get value or disappointment. The benchmark shows that today's best AI agents reach only 56 percent effectiveness on realistic tasks, meaning organizations betting on AI for complex customer work are likely to see failed projects and wasted investment until these systems learn to ask better questions and reason through messy, incomplete information.