GPUs, TPUs, and NPUs: What AI Hardware Does and Why It Matters

Every neural network is a stack of matrix multiplications. Hardware that runs those multiplications faster, cheaper, or with less power wins. GPUs, TPUs, and NPUs all attack the same math with different trade-offs in flexibility, throughput, and energy use. Pick the wrong one and you either overpay for idle silicon or wait days for a training run that should take hours.

Quick Takeaways

• GPUs are the general-purpose standard. They dominate training and offer the broadest software ecosystem (CUDA, ROCm, PyTorch).
• TPUs are Google’s custom ASICs built around systolic arrays. They excel at large-scale training and high-throughput inference inside Google Cloud.
• NPUs are low-power accelerators embedded in phones, laptops, and edge devices. They run INT8/INT4 inference locally at a few watts.
• Memory bandwidth, not raw FLOPS, is usually the bottleneck for LLM inference.

Specs vary by generation and vendor. Verify current figures before purchasing.

Why AI Needs Specialized Hardware

A CPU has a handful of powerful cores optimized for sequential logic and branching. A transformer layer needs the opposite: millions of identical multiply-accumulate (MAC) operations running at once.

Consider a single forward pass through a 70B-parameter model. At FP16 precision, the weights alone occupy about 140 GB. Each generated token requires reading most of those weights from memory and performing roughly 140 GFLOPs of compute. CPUs cannot feed that pipeline fast enough.

Specialized accelerators solve three problems:

1. Parallelism: thousands of arithmetic units working simultaneously.
2. Memory bandwidth: high-bandwidth memory (HBM) stacked next to the compute die.
3. Low-precision math: formats like BF16, FP8, INT8, and INT4 that cut memory and energy per operation.

GPUs: The General-Purpose Workhorse

Architecture

A GPU contains many streaming multiprocessors (SMs). Each SM holds CUDA cores for general math and Tensor Cores for matrix operations. Tensor Cores execute small matrix multiply-accumulate tiles (for example, 16×16) in a single instruction.

Key architectural elements:

• Tensor Cores: support FP16, BF16, FP8, and on newer generations FP4.
• HBM3/HBM3e memory: delivers multiple TB/s of bandwidth on data-center parts.
• NVLink / Infinity Fabric: high-speed interconnects for multi-GPU scaling, often 600-900 GB/s per GPU.
• Programmability: custom kernels via CUDA, Triton, or HIP.

Strengths

• Runs almost any model architecture, including new research ideas, without waiting for compiler support.
• Mature libraries: cuDNN, cuBLAS, FlashAttention, vLLM, TensorRT-LLM.
• Available on every major cloud and in on-premises clusters.

Weaknesses

• High power draw and cooling requirements.
• Supply constraints and premium pricing for flagship data-center parts.
• General-purpose overhead means lower efficiency than a purpose-built ASIC for fixed workloads.

Major Vendors

NVIDIA leads with its data-center line (Hopper, Blackwell, and successors). AMD competes with its Instinct series and the open ROCm stack. Intel offers Gaudi accelerators with a distinct architecture.

TPUs: Google’s Matrix Engine

Architecture

A TPU (Tensor Processing Unit) is an application-specific integrated circuit designed around the systolic array. In a systolic array, data flows rhythmically through a grid of MAC units. Each unit passes its result to its neighbor, so intermediate values are never written back to memory.

This design delivers two benefits:

• Fewer memory accesses, which cuts energy per operation.
• Predictable, deterministic execution, which simplifies scheduling.

TPUs are connected in pods using a dedicated high-speed inter-chip interconnect (ICI) in a 2D or 3D torus topology. Large pods link thousands of chips into one training system.

Strengths

• Strong performance per watt for dense matrix workloads.
• Excellent scaling for large training runs when code is written in JAX or compiled through XLA.
• Competitive cost for sustained, large-batch workloads on Google Cloud.

Weaknesses

• Vendor lock-in: available only through Google Cloud.
• Dynamic control flow and irregular operations can be harder to compile efficiently.
• Smaller community and fewer third-party kernels than CUDA.

Quick Start: Detecting a TPU in JAX

# pip install "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
import jax
import jax.numpy as jnp

print(jax.devices())  # Lists available TPU cores

@jax.jit  # Compile via XLA for the accelerator
def matmul(a, b):
    return jnp.dot(a, b)

key = jax.random.PRNGKey(0)
a = jax.random.normal(key, (4096, 4096), dtype=jnp.bfloat16)  # BF16 is TPU-native
b = jax.random.normal(key, (4096, 4096), dtype=jnp.bfloat16)

result = matmul(a, b).block_until_ready()
print(result.shape)

NPUs: Intelligence at the Edge

Architecture

An NPU (Neural Processing Unit) is a small, fixed-function accelerator integrated into a system-on-chip (SoC). It targets inference only, with heavy emphasis on INT8 and INT4 quantized models. Most NPUs share system memory with the CPU and GPU, which keeps cost and power low but limits bandwidth.

Performance is typically quoted in TOPS (trillions of operations per second). Be careful: TOPS usually refers to INT8 throughput and says little about real-world LLM speed, which depends on memory bandwidth.

Where You Find Them

• Smartphones: Apple Neural Engine, Qualcomm Hexagon, Google Tensor.
• Laptops: Intel Core Ultra, AMD Ryzen AI, Qualcomm Snapdragon X series, Apple M-series. Microsoft’s Copilot+ PC branding set a 40+ TOPS NPU baseline.
• Edge and IoT: cameras, cars, industrial sensors.

Strengths

• Extreme efficiency: running always-on tasks (voice wake words, background blur, translation) with minimal battery impact.
• Privacy: data stays on the device.
• Low latency: no network round trip.

Weaknesses

• Limited model size. On-device LLMs usually top out at 1B-8B parameters with 4-bit quantization.
• Fragmented software tooling across vendors.
• Not suited for training.

Quick Start: Running a Model on an NPU with ONNX Runtime

# pip install onnxruntime-directml   (Windows) or the vendor-specific execution provider
import onnxruntime as ort
import numpy as np

# Execution provider varies: QNNExecutionProvider (Qualcomm), OpenVINOExecutionProvider (Intel)
session = ort.InferenceSession(
    "model_int8.onnx",
    providers=["QNNExecutionProvider", "CPUExecutionProvider"],  # Falls back to CPU
)

input_name = session.get_inputs()[0].name
dummy = np.random.rand(1, 3, 224, 224).astype(np.float32)
outputs = session.run(None, {input_name: dummy})
print(outputs[0].shape)

Head-to-Head: Architecture and Performance Trade-Offs

The Real Bottleneck: Memory Bandwidth

Raw compute rarely limits LLM inference. Memory bandwidth does.

During autoregressive decoding, the model generates one token at a time. Each token requires streaming the entire set of weights from memory. A useful rule of thumb:

Max tokens/sec ≈ Memory Bandwidth (GB/s) / Model Size (GB)

For a 7B model in 4-bit quantization (about 3.5 GB):

These are theoretical ceilings. Real throughput is lower due to KV-cache reads, kernel overhead, and batching behavior.

Batching changes the equation. Serving many requests at once amortizes each weight read across multiple tokens, shifting the workload from memory-bound toward compute-bound. This is why data-center accelerators with high FLOPS and high bandwidth dominate serving at scale.

Quantization: The Great Equalizer

Quantization reduces numerical precision to shrink memory use and speed up inference.

Quantization is what makes NPUs viable for language models. It is also why a single high-memory GPU can now serve models that once required a multi-GPU node.

Example: Loading a 4-Bit Model with Hugging Face

# pip install transformers accelerate bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # NormalFloat4 quantization
    bnb_4bit_compute_dtype=torch.bfloat16 # Compute in BF16 for stability
)

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=config,
    device_map="auto",                    # Place layers across available devices
)

inputs = tokenizer("Explain systolic arrays in one sentence.", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=60, temperature=0.2)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Practical Workflows and Real-World Use Cases

Scenario 1: Startup Fine-Tuning an Open-Source LLM

Goal: Fine-tune an 8B model on 50,000 support tickets.

• Recommended hardware: a single GPU with 40-80 GB of memory.
• Technique: LoRA/QLoRA reduces trainable parameters to under 1% of the total, so the job fits on one card.
• Why not TPU or NPU: the GPU ecosystem offers the fastest path from idea to result, and NPUs cannot train.

Scenario 2: Enterprise Serving a High-Traffic Chatbot

Goal: Serve thousands of concurrent users with low latency.

• Recommended hardware: a GPU cluster with vLLM or TensorRT-LLM, or TPUs if the stack already runs on Google Cloud.
• Key techniques: continuous batching, PagedAttention for KV-cache efficiency, and FP8 quantization.
• Metric to track: cost per 1M tokens and p95 latency, not peak FLOPS.

Scenario 3: Privacy-First Mobile or Laptop Assistant

Goal: Summarize documents locally with no cloud calls.

• Recommended hardware: an NPU-equipped device running a 3B-8B model at INT4.
• Benefit: zero data egress, offline operation, and no per-token fees.
• Constraint: keep context windows modest (for example 4K-8K tokens), since KV-cache memory grows with context length.

Scenario 4: Foundation Model Pre-Training

Goal: Train a model from scratch on trillions of tokens.

• Recommended hardware: thousands of GPUs with InfiniBand/NVLink, or large TPU pods.
• Key factors: interconnect bandwidth, fault tolerance, and software maturity at scale. At this size, a single percentage point of utilization (MFU) is worth millions of dollars.

How to Choose: A Decision Framework

Benchmark before committing. Use your actual model, batch size, and sequence length. Vendor peak-FLOPS figures rarely predict real workload performance.

Common Mistakes to Avoid

• Comparing TOPS across devices. Different vendors measure at different precisions.
• Ignoring memory capacity. A model that does not fit in memory will not run, regardless of FLOPS.
• Overlooking software maturity. An unsupported operator can erase any hardware advantage.
• Skipping quantization. Moving from FP16 to INT8 or FP8 often doubles effective throughput.
• Ignoring total cost. Power, cooling, networking, and engineering time often exceed chip cost.

Frequently Asked Questions

What is the main difference between a GPU, TPU, and NPU?

A GPU is a flexible, massively parallel processor used for both training and inference. A TPU is Google’s custom ASIC optimized for matrix math. An NPU is a low-power chip inside consumer devices that runs quantized inference locally.

Is a TPU faster than a GPU?

It depends on the workload. TPUs can deliver strong performance per watt and cost efficiency for large, dense training jobs written in JAX or TensorFlow. GPUs offer broader model support and a larger software ecosystem. Benchmark your own workload to decide.

Can an NPU run large language models?

Yes, but only small ones. Current NPUs typically run 1B-8B parameter models at 4-bit precision. Larger models exceed the memory capacity and bandwidth of consumer devices.

Why are GPUs used for AI instead of CPUs?

AI workloads rely on enormous numbers of parallel matrix multiplications. GPUs contain thousands of arithmetic units and high-bandwidth memory built for exactly that pattern, while CPUs prioritize sequential, low-latency tasks.