Ai2 researchers introduced StateBench, a new benchmark designed to evaluate how well AI models track variable assignments and entity states in long sequences. The evaluation framework tests performance across four distinct difficulty levels, focusing specifically on the ability to maintain state information over extended inputs.
The benchmark measures performance on sequences up to 64,000 tokens, covering both finite and infinite state tracking scenarios. Frontier models including GPT-6 Sol, GPT-6 Luna, Claude Opus 5.5, and Gemini 3.8 Flash struggled with ‘general finite’ problems as the input length increased. Accuracy for these models dropped to 13 percent or lower on the longest problems.
StateBench reveals that Gated DeltaNet and hybrid models can generalize much farther than standard transformers when trained directly on the data. A Gated DeltaNet variant with flexible memory updates generalized to roughly 10 times the training sequence length, while transformers could not learn the hardest state-tracking level even with targeted training. This highlights a critical limitation in current transformer architectures for maintaining context over long durations.
