Puzzles and games have been a benchmark for AI progress since Arthur Samuel popularized the term "machine learning" in 1959 with an algorithm that learned to play checkers. Chess, Go, and now crossword-style word games have all served as testing grounds for measuring how far artificial intelligence has come—and where it still falls short.

What Happened

In late 2024, Columbia University researchers found that even the best AI models could solve only about 18% of New York Times Connections puzzles. By early 2025, some models had improved to near-perfect performance on those same word puzzles. However, a study published in 2024 by Google and the University of Illinois Urbana-Champaign revealed persistent vulnerabilities: when trained and tested on variations of Knights and Knaves logic problems—where characters either always tell truths or always lie—models tended to rely on memorized patterns rather than adapting their reasoning. The researchers found that models often whizzed past key differences in puzzle variants, responding with training data rather than solving the problem at hand.

Why It Matters

The gap between AI performance and human cognition reveals critical insights for developers building applications that depend on reliable reasoning. Spatial reasoning tasks like mental rotation puzzles still trip up language models despite their ability to process visual inputs—a significant limitation for agents meant to navigate physical environments or assist in architecture, engineering, or robotics. Similarly, the SimpleBench evaluation shows that problems resembling training examples cause even top-tier models to stumble where humans spot the underlying trick. For enterprises deploying AI in customer service, legal analysis, or diagnostic tools, these failure modes represent real-world risk. Understanding where current models break down helps developers set appropriate expectations and design systems that compensate for known weaknesses.

The Bottom Line

AI has made rapid strides on puzzle-based benchmarks like ARC-AGI, but significant gaps remain. Spatial reasoning, visual manipulation, and adaptation to novel problem variants continue to challenge even frontier models—areas where human cognition still holds advantages.