Ai2 has introduced XYBench, a new research benchmark designed to evaluate how artificial intelligence models handle misconceptions in user queries. The test suite contains 8,115 questions and focuses on whether an AI identifies an implicit goal versus a literal request. The developers hope this tool will help improve the reasoning capabilities of large language models.
The evaluation was conducted on several leading models including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro. Researchers from Ai2, Carnegie Mellon, NVIDIA, and Johns Hopkins collaborated on the project. The dataset for the benchmark was sourced from Stack Overflow, Stack Exchange, WikiHow, and manually curated content.
The results indicate that many models prioritize the literal request over the underlying goal, which can lead to incorrect or unhelpful answers. Claude Sonnet 4.5 was used as the automated judge for this assessment. This finding suggests a need for better instruction following in AI systems. Developers should use this benchmark to train models that understand user intent more accurately.
