In the Age of Agentic AI, CPUs Matter More Than Ever

For years, AI infrastructure and discourse have revolved around GPUs. Training large models requires running billions of calculations in parallel, and specialized accelerators are designed to do that efficiently. As a result, AI progress has often been measured in terms of GPU scale.

Now the industry is moving into a new phase. As models move into actual applications deployed for working use cases, the demands inside the data center are evolving. The distinction between AI and general-purpose workloads is fading. The growth is now in inference—running AI continuously in real-world applications. 

That inference is increasingly agentic: Instead of responding to a single prompt, agentic systems involve many AI agents that plan tasks, retrieve information, interact with databases and adapt in real time. They operate continuously, placing new demands on the infrastructure that powers AI. Agentic AI systems rely on web services, networking, data pipelines, storage and orchestration layers, creating a convergence between AI inference and general-purpose enterprise workloads. In this new environment, AI performance increasingly depends on the efficiency of the broader compute stack, not just the accelerator.

“The industry has spent years scaling model compute,” says Mohamed Awad, EVP of Cloud AI at Arm (NASDAQ: ARM). “But inference is where AI creates value—where models interact with users, drive decisions and operate continuously in production. That shifts the focus to the systems layer.”

That shift is placing more importance on a foundational component of the AI stack: the CPU. It manages task coordination, data movement, networking and system orchestration. It also sustains continuous inference and ensures accelerators remain fed and fully utilized for the right tasks. As agentic AI scales, so does the demand for efficient CPUs that can support growth with performance and efficiency.

The server CPU market is projected to grow from $26 billion in 2025 to roughly $60 billion by 2030—evidence of a structural shift in how AI systems are being designed and deployed.

The architectural change becomes clearer when comparing traditional inference to emerging agentic systems.

Traditional large language model (LLM) inference workloads are relatively well defined. A request enters the system, the model executes forward passes on an accelerator to generate tokens and the response is returned.  

Agentic systems, on the other hand, break down complex objectives into sub-tasks. They coordinate multiple models, call external tools, access data resources, retrieve data, manage context, break tasks down and adapt based on immediate results. The model forward pass remains important, but is embedded within a broader workflow.

For example, orchestration layers handle continuous scheduling, resource allocation and coordination across components. The workload becomes less about the peak mathematical throughput alone and more about efficiently managing complex control flow across the system.

Every AI deployment relies on a CPU operating alongside accelerators. The CPU schedules work to GPUs, helps manage memory and key-value caches, handles pre and post-processing and supports interactions with vector databases and storage systems. As agent-based inference scales, the scope and intensity of these coordination responsibilities expand, reinforcing the CPU’s role as the system’s control layer.

Because agentic AI systems typically run as continuous services, maintaining state and coordinating tasks over extended periods, their infrastructure must support sustained, always-on workloads.

At the same time, AI hardware is becoming denser. Modern rack-scale systems pack far more accelerators and host processors into a single footprint than earlier server designs. NVIDIA’s latest rack architectures, for example, integrate dozens of Arm-based CPUs alongside GPUs, while AWS’s newest Trainium systems rely on Arm-based Graviton processors to coordinate large accelerator clusters.

As accelerator density rises and additional data processing units are layered into each rack, the coordination burden rises accordingly. But data center power budgets don’t expand at the same rate.

Scaling AI is no longer just about adding more accelerators; it depends on delivering more output within fixed energy constraints. As racks incorporate more host processors, CPU efficiency directly affects total system power, cost and scalability. Performance per watt is becoming as critical as raw accelerator throughput.

That shift comes as the balance of AI workloads is changing. According to Bloomberg Intelligence, the inference market is expected to surpass training by 2029, driven by the rise of reasoning models and AI agents. As inference becomes the dominant workload, the efficiency of the control layer becomes increasingly central to infrastructure design.

The market response reflects this shift. Leading hyperscalers are rolling out new generations of Arm-based server CPUs with higher core counts to support the growing orchestration demands of agentic workloads while maintaining performance per watt. AWS’s latest Graviton processor scales to 192 cores, Microsoft’s Cobalt 200 to 132 cores and NVIDIA’s upcoming Vera CPU increases core density over prior generations—all increasing performance without compromising efficiency.

Together, these developments indicate a clear trajectory: AI and general-purpose workloads are converging as agentic AI drives sustained inference across the data center. CPU performance and efficiency will become critical differentiators in orchestrating AI systems.

Bloomberg Intelligence research has found that Arm is increasingly emerging as the preferred control-plane architecture in AI infrastructure, citing its power efficiency and scalable core designs as hyperscalers expand adoption. Arm Neoverse-based CPUs have now surpassed one billion deployed data center cores, underscoring how central efficient general-purpose compute has become in modern AI systems.

Accelerators remain essential for executing model math. Yet as AI systems become more distributed, persistent and interactive, the coordination layer increasingly shapes overall performance and cost. Delivering AI at scale depends not only on how fast models compute, but also on how efficiently the system runs over time.

In that context, CPUs are assuming a larger role in AI architecture—one that is likely to expand.

“The next phase of AI isn’t just about building smarter models – it’s about building smarter systems,” Awad says. “The efficiency of the systems layer will increasingly determine how scalable and sustainable AI becomes.”