More than four out of five (83%) organizations say they need to upgrade their infrastructure before they can fully support agentic AI. That’s a striking number, especially considering how quickly enterprises are trying to roll out AI across the business.
The excitement around AI hasn’t slowed down, but for many organizations, the infrastructure is still playing catch-up.
Google explores that gap in its new State of AI Infrastructure report, which surveyed 1,400 IT leaders around the world. The findings offer a snapshot of where enterprise AI stands today and where organizations are investing as AI workloads become more complex and increasingly inference-driven.
“For years, enterprise AI has been synonymous with conversational AI — the customer service bots and digital assistants we interact with every day,” wrote Drew Bradstock, Senior Director, Product, Orchestration & Kubernetes.
(Credits: Google Cloud state-of-ai-infrastructure-report-overview)
“But today, the market has shifted. We’ve officially moved from AI that answers through simple chats, to AI that takes action, automated workflows, and executes complex tasks on its own. While this unlocks entirely new use cases, there’s a catch: it places significant stress on the underlying infrastructure we’ve relied on in the past.”
This is where things start to change. Most enterprise infrastructure wasn’t built for AI systems that are constantly reasoning and making decisions. It was built for people using software. Agentic AI flips around that equation.
What that means is that agentic AI demands a new scale of execution. Organizations are spending less time thinking about training models and more time figuring out how to run them efficiently at scale. The report found that inference now accounts for 47% of AI workloads. This is followed by training at 28% and model optimization at 16%.
Inference may be taking over, but we must keep in mind that it still depends on everything around it. For example, AI agents still need to connect to CRMs, ERPs, databases, and other business systems to get work done. That’s easier said than done. The report found that legacy APIs and data sources remain the biggest infrastructure gap – followed by vector databases and security for multi-system access.
“Agentic workloads introduce a new level of scale, where a single prompt can trigger hundreds of downstream actions, requiring massive context windows to be held in memory,” explained Bradstock. “Trying to run these continuous reasoning loops on legacy architecture is financially unsustainable.”
The report reveals that 62% of leaders are seeing a significant inference tax driven by data egress fees, storage bloat, and idle specialized hardware. As many as 81% cite operational complexity as a hidden cost of scaling AI.
(Credits: Google Cloud state-of-ai-infrastructure-report-overview)
The report also spends quite a bit of time on what happens after organizations start deploying AI agents. Google refers to it as agent sprawl. Instead of managing one or two AI applications, enterprises could end up with hundreds or even thousands of agents accessing different systems across the business. That makes visibility and governance a lot harder, especially as agents are given more autonomy.
Another point that stood out is data. AI agents can’t do much if the information they need is scattered across different databases and applications. Google argues that organizations will need a more unified data layer so agents can find and use information without relying on endless custom integrations or duplicated data.
The report also points out something that probably wasn’t on many people’s radar a couple of years ago: energy. Bradstock wrote, “Energy consumption used to be a sustainability metric reserved for annual reports. Today, it plays a crucial operational role.”
The survey found that 91% of technology leaders now consider power consumption when choosing hardware. AI infrastructure isn’t just becoming a compute problem anymore. It’s becoming a power problem too.
In a way, the report marks a change in the conversation around AI. A year or two ago, most of the focus was on the models. Today, the questions are different. Can the infrastructure keep up? Can it support hundreds of AI agents? Can it run efficiently? Can it do all of that without costs spiraling? Those are the problems enterprises seem to be spending more time thinking about now.
Google’s answer isn’t necessarily to buy more hardware. It’s to rethink the entire stack. Throughout the report, the company argues that compute, storage, networking, governance, and data all need to work together if organizations want to move from AI pilots to production. Looking at any one of those areas in isolation is becoming harder as AI workloads continue to grow.
The post Inside Google’s New AI Infrastructure Report appeared first on AIwire.
