(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable
Keeping AI honest by making humans check its work
When researchers let AI systems work unsupervised, they fail catastrophically 72% of the time. A structured approach that keeps humans in control—where AI suggests ideas but humans execute all data work and make final calls—cuts that failure rate to 16%, even using the exact same AI model. The gains were largest when studying unfamiliar datasets, suggesting this human-AI partnership works best on novel research problems.
As universities and companies race to use AI for research, blindly trusting AI outputs can publish false findings that waste resources and mislead policy. This framework shows that reliability doesn't require better AI alone—it requires better workflow design, with specific checkpoints where human judgment stops bad analyses before they reach publication. The method is practical enough to deploy today with existing tools.