StakeBench: Evaluating Language Understanding Grounded in Market Commitment
Testing AI's ability to understand what money actually says about beliefs
Researchers created StakeBench, a new test for AI language understanding based on real financial commitments rather than human opinions. They linked nearly 561,000 comments from prediction markets to actual trades and betting positions, then measured whether 15 large language models could identify what people had put money behind. Most models performed poorly—detecting the correct position only about half the time, and completely failing at predicting future trades or collective market movements, even when they were very large.
Financial institutions and traders increasingly rely on AI to interpret market commentary and news. This benchmark reveals that today's best models can't reliably extract the actual beliefs people are willing to bet on, which means systems used to inform real investment decisions are systematically misunderstanding what market participants truly think. The findings also suggest that simply making models bigger or training them on finance data doesn't solve the problem.