A study from researchers at UC Berkeley evaluated AI agents' ability to complete autonomous shopping transactions and found significant reliability gaps across multiple platforms, according to reporting by The Decoder.
What Happened
The research team—affiliated with Berkeley's Center for Human-Compatible AI—tested two frontier models, Claude (Anthropic) and ChatGPT (OpenAI), on 150 real-world e-commerce tasks spanning checkout processes, account credential management, and responses to dynamic pricing or inventory changes. The agents failed 41% of tasks overall, with success rates varying by platform: Amazon at 38%, Walmart at 28%, and Target at 18%. Specific failure modes included inability to handle CAPTCHA challenges during account creation, incorrect handling of multi-item discount codes, and freezing when confronted with unexpected page layouts or pop-ups. The study notes that agents performed better on tasks requiring single-step navigation but degraded significantly when transactions required cross-page context or error recovery.
Why It Matters
Lead researcher Professor Stuart Russell told The Decoder: 'The gap between what these systems can do in controlled benchmarks and real retail environments remains substantial.' For developers building agentic shopping tools, the findings highlight that capability demos emphasizing single-task accuracy obscure multi-step transaction reliability. Businesses exploring AI-powered purchasing assistants face deployment challenges because current systems cannot consistently navigate the friction points of online retail—including dynamic pricing, inventory fluctuations, and checkout edge cases—without human oversight.
The Bottom Line
The Berkeley evaluation demonstrates that autonomous shopping remains an unsolved problem for current frontier models. While Claude and ChatGPT achieved near-complete accuracy on individual steps like product searches or cart additions, end-to-end transaction success rates fell below 60% across major retail platforms. Users relying on such tools for hands-off purchasing may encounter frequent failures or require constant monitoring to complete transactions successfully.