AI Chips in 2026: NVIDIA, AMD, Google TPUs, and Custom Silicon Compared

Memory, not raw FLOPS, now decides which AI chip wins. In 2026, every major accelerator moved to HBM4 or large HBM3e stacks, rack-scale designs replaced the single GPU as the unit of purchase, and hyperscalers started shipping serious custom silicon. This guide compares the leading platforms on the numbers that affect real deployments: capacity, bandwidth, low-precision throughput, interconnect, software, and access.

Quick Takeaways

• NVIDIA Rubin leads on peak compute and bandwidth: 288 GB HBM4, 22 TB/s, and 50 PFLOPS NVFP4 inference per GPU.
• AMD MI455X leads on capacity: 432 GB HBM4 per GPU and 31 TB per 72-GPU Helios rack, with an open-standards fabric.
• Google TPUs and AWS Trainium3 win on cost and scale inside their own clouds. Maia 200 and MTIA mostly serve their owners.
• Vendor FLOPS figures use different precisions and sparsity assumptions. Compare on your own model, batch size, and context length.

*AMD and its partners publish figures between 19.6 and 23.3 TB/s depending on the document.

How to Read an AI Chip Spec Sheet

Peak numbers mislead if you ignore four variables.

Precision. A “50 PFLOPS” claim at FP4 is not comparable to a “4.6 PFLOPS” claim at FP8. Halving the bit width roughly doubles peak throughput, but it also changes model quality. NVIDIA’s NVFP4 and the OCP MXFP4 format are different implementations of 4-bit floating point.

Memory bandwidth. Autoregressive decoding reads the full set of active weights and the KV cache for every generated token. For large models, decode speed is bandwidth-bound, not compute-bound.

Memory capacity. Capacity determines how many chips one model replica needs. Fewer chips per replica means less inter-chip communication and lower latency.

Scale-up domain. NVLink, UALink, and Google’s ICI define how many chips act as one machine. Mixture-of-Experts (MoE) models send tokens between experts constantly, so this fabric directly limits throughput.

NVIDIA Rubin and Vera Rubin NVL72

Rubin succeeds Blackwell in H2 2026. NVIDIA reports that it entered full production in early 2026 and ships to hyperscalers and neoclouds this half.

Rubin GPU specifications

• Process: TSMC 3nm, dual-die design, 336 billion transistors
• Memory: 288 GB HBM4 across 8 stacks at 22 TB/s
• Compute: 50 PFLOPS NVFP4 inference, 35 PFLOPS NVFP4 training
• Interconnect: NVLink 6 at 3.6 TB/s per GPU

Bandwidth is the biggest jump. At 22 TB/s, Rubin offers about 2.8x the 8 TB/s of Blackwell B200. HBM4 doubles the per-stack interface width to 2,048 bits.

The rack is the product

The Vera Rubin NVL72 combines 72 Rubin GPUs and 36 Vera CPUs (88 custom Arm cores each). NVIDIA rates it at:

• 3.6 EFLOPS NVFP4 inference and 2.5 EFLOPS training
• 20.7 TB HBM4 and 260 TB/s aggregate NVLink bandwidth
• Roughly 190-230 kW rack power, which requires liquid cooling

NVIDIA claims up to 10x lower cost per token than Blackwell. Treat that as a best-case figure for specific workloads, since it depends on NVFP4 adoption and heavy software tuning.

Strengths: the mature CUDA ecosystem, the fastest scale-up fabric, and the widest availability across clouds.
Trade-offs: premium pricing, tight HBM4 supply, and the lowest capacity per GPU among the two merchant-silicon flagships.

AMD Instinct MI455X and the Helios Rack

AMD’s MI400 series is its first serious bid at rack-scale design. The MI455X uses the CDNA 5 architecture.

MI455X specifications

• Memory: 432 GB HBM4 in 12 stacks, about 50% more than Rubin
• Bandwidth: 19.6 TB/s in AMD’s rack brochure (some partner pages list 23.3 TB/s)
• Compute: up to 40 PFLOPS FP4
• Silicon: about 320 billion transistors across 2nm and 3nm chiplets

Helios rack specifications

Helios packs 72 MI455X GPUs and 18 EPYC “Venice” CPUs into a double-wide Open Rack Wide chassis, a standard Meta contributed to the Open Compute Project. AMD rates it at 2.9 EFLOPS FP4, 1.4 EFLOPS FP8, and 31 TB HBM4. Scale-up uses UALink over Infinity Fabric. Scale-out uses Ultra Ethernet.

Strengths: capacity per GPU, an open networking stack, and ROCm support that has matured in mainstream PyTorch and vLLM.
Trade-offs: a younger rack-scale track record, and kernel-level tuning that often lags CUDA.

Google TPUs: Ironwood Today, TPU 8 Next

Ironwood (TPU v7)

Google made Ironwood generally available in April 2026.

• 4.6 PFLOPS peak FP8 per chip, about 4x Trillium
• 192 GB HBM3e at roughly 7.4 TB/s
• 9,216-chip superpods delivering 42.5 EFLOPS FP8, with about 1.77 PB of shared HBM

Per chip, Ironwood sits close to Blackwell B200 on FP8 throughput and HBM capacity. It lacks Rubin’s 4-bit throughput and HBM4 bandwidth. Its strength is pod scale: thousands of chips on a 3D torus interconnect, programmed through JAX/XLA or PyTorch.

TPU 8t and TPU 8i

Google split its eighth generation in two:

The 8i’s large SRAM keeps parts of the KV cache on-die, which targets long-context reasoning and MoE serving. Reported availability for the TPU 8 generation is late 2027, so plan around Ironwood for now.

Custom Silicon: Trainium3, Maia 200, and MTIA

Hyperscalers design these chips for their own workloads, which changes the buying calculus.

Trainium3 requires the Neuron SDK, which ties your code to AWS. A Trainium3 UltraServer links up to 144 chips, and both major labs and enterprises use it for training and inference.

Maia 200 targets inference at about 750 W, with roughly 10 PFLOPS FP4, 272 MB of on-chip SRAM, and Ethernet-based scale-up. It serves Copilot and Azure-hosted OpenAI traffic.

MTIA is an inference-first roadmap. Meta’s published figures show HBM bandwidth growing 4.5x and compute 25x from the 300 to the 500 generation, at module power up to 1,700 W.

Head-to-Head: What Differs in Practice

Practical Workflow: Size Memory Before You Pick a Chip

Start with memory, because it eliminates options fastest.

Weights: parameters × bytes per parameter (FP16 = 2, FP8 = 1, FP4 = 0.5)
KV cache per sequence: 2 × layers × KV heads × head dim × sequence length × bytes

def kv_cache_gb(layers, kv_heads, head_dim, seq_len, bytes_per=2, batch=1):
    # Keys and values -> factor of 2
    return 2 * layers * kv_heads * head_dim * seq_len * bytes_per * batch / 1e9

def weights_gb(params_b, bytes_per):
    return params_b * bytes_per  # params in billions

# Example: 70B-class dense model, GQA with 8 KV heads, 128k context
w = weights_gb(70, 1)                       # FP8 weights -> 70 GB
kv = kv_cache_gb(80, 8, 128, 131_072)       # FP16 KV cache -> ~43 GB per sequence
print(f"Weights: {w:.0f} GB | KV per 128k sequence: {kv:.0f} GB")

The output is 70 GB of weights plus roughly 43 GB of KV cache per 128k-token sequence. Eight concurrent long-context users need about 344 GB for cache alone. That exceeds one Rubin’s 288 GB and fits within one MI455X’s 432 GB once you add the weights, which is why capacity can beat bandwidth for long-context serving.

Serve it with vLLM

The same command works on CUDA and ROCm builds. Adjust the tensor-parallel size to your chip count:

pip install vllm
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --quantization fp8 \
  --gpu-memory-utilization 0.90

Check your hardware from Python

# NVIDIA or AMD (ROCm builds reuse the torch.cuda API)
import torch
print(torch.cuda.get_device_name(0), torch.cuda.get_device_properties(0).total_memory / 1e9)

# Google TPU
import jax
print(jax.devices())

Real-World Scenarios

Frontier training. Choose Rubin NVL72 for the NVLink fabric and CUDA tooling, or TPU pods if you already run on Google Cloud. Training at this scale punishes software friction more than it rewards a lower chip price.

High-concurrency chat inference. Bandwidth drives tokens per second per user. Rubin’s 22 TB/s fits latency-sensitive serving. Maia 200 and TPU 8i target the same cost-per-token problem inside their clouds.

Long-context and RAG workloads. KV caches dominate memory. MI455X’s 432 GB lets one GPU hold larger models and more concurrent sessions, which cuts the cross-GPU traffic that adds latency.

Cost-sensitive batch jobs on AWS. Trainium3 can lower training and inference costs if your team can absorb the Neuron SDK migration. Run a pilot on one model before committing.

Multi-cloud strategy. Prefer chips that run PyTorch and vLLM with minimal changes. That favors NVIDIA and AMD, with TPUs reachable through PyTorch/XLA at some engineering cost.

Strategic FAQ

Which AI chip is fastest in 2026?

NVIDIA Rubin has the highest published per-GPU figures: 50 PFLOPS NVFP4 and 22 TB/s HBM4 bandwidth. AMD’s MI455X offers more memory (432 GB), and Google’s pods scale to thousands of chips. Real speed depends on your model, precision, and software stack.

Is AMD MI455X better than NVIDIA Rubin?

It depends on the workload. MI455X provides 50% more HBM4 per GPU, which helps with large models and long contexts. Rubin leads on peak bandwidth, 4-bit compute, and software maturity. Benchmark your own model before deciding.

Can I rent Google TPUs, Trainium3, or Maia 200?

You can rent Ironwood on Google Cloud and Trainium3 on AWS EC2. Maia 200 and MTIA are mainly internal to Microsoft and Meta, with no general public rental option.

Do custom AI chips replace GPUs?

Not broadly. Custom chips cut costs for large, stable inference workloads inside one cloud. GPUs remain the default for research, frontier training, and portability across providers.