Startups Seek Alternatives as Transformer Architecture Hits Limits in LLMs

Nine years after Google researchers introduced the transformer architecture, a wave of startups is pushing to move beyond the technology that underpins every major large language model on the market.

What Happened

The transformer architecture powers all mainstream LLMs through dense attention—a mechanism that compares every word or token with every other via multiplication. As models grow more capable, this approach has become increasingly costly and inefficient. According to OpenAI's president Greg Brockman, the company is set to spend $50 billion on computing this year, while the International Energy Agency predicts data center electricity consumption will double by 2030. Transformers also struggle with expanding context windows and handling the additional data reasoning models generate through chain-of-thought scratch pads. At least two startups are pursuing alternatives: Subquadratic, a Miami-based firm that claims it has developed the first sparse attention mechanism rivaling top mainstream LLMs on search and coding tasks; and Manifest AI, based in San Francisco, which is replacing traditional attention with a system called power retention to manage data growth more efficiently.

Why It Matters

The transformer bottleneck creates both risk and opportunity. For developers and enterprises deploying LLMs at scale, compute costs and energy consumption are becoming limiting factors. If sparse attention or alternative architectures can deliver comparable performance with fewer multiplications, it could lower barriers to entry and reshape competitive dynamics among AI labs. The International Energy Agency's electricity projections highlight the environmental stakes of continued scaling under current paradigms.

The Bottom Line

Subquadratic reports thousands have signed up to its waitlist and plans a wider release soon. Whether sparse attention mechanisms can genuinely match dense attention across all tasks remains an open question, according to some industry observers. The outcome will shape which companies lead the next generation of language models.