Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
How AI picks the right meaning before scientists do math
When scientists need to compare experimental groups, the way they define what counts as the same—a culture, a treatment, a reference point—changes everything. Researchers tested an AI component called Jev that makes these semantic choices before calculations happen, comparing it against five other approaches across ten real scientific scenarios. Jev matched the best alternatives at picking the right definitions, but the team found that wrong choices in one area could silently change the numbers downstream without breaking the final answer—a hidden risk.
Scientists rely on automated workflows to process data faster, but if an AI picks the wrong interpretation of what's being measured, the final numbers could be wrong in ways that don't trigger any warning. This work shows that checking only the final answer isn't enough—you have to audit the semantic choices and intermediate values the system actually used, or errors can slip through undetected.