Best Hardware for Running Local AI in 2026: GPUs, Macs, and Mini PCs

Memory, not compute, decides what you can run locally in 2026. A model that fits in fast memory feels instant. A model that spills over crawls or fails to load. The hardware market has also split into three shapes: fast but small (discrete GPUs), big but slower (128 GB unified-memory boxes), and big and fast but expensive (Apple’s Ultra tier and workstation cards). A DRAM and GDDR7 shortage has also pushed some prices far past launch MSRP. Check the numbers below against live listings before you buy.

Quick Takeaways

• Model fits in 32 GB? An NVIDIA GPU is the fastest option per dollar of capability, but the RTX 5090 is badly overpriced right now. A used RTX 3090 (24 GB) is the sane alternative.
• Need 70B-120B models? Choose a 128 GB unified-memory machine: Mac Studio, AMD Strix Halo, or NVIDIA DGX Spark.
• Fastest single-user large-model speed: Mac Studio M5 Ultra (up to 512 GB at ~1.2 TB/s).
• Cheapest path to 128 GB: Strix Halo mini PCs, with slower prompt processing as the trade-off.

Prices fluctuate weekly. Treat every figure here as a snapshot and confirm before buying.

The Two Numbers That Decide Everything

1. Memory capacity: will the model fit?

Estimate required memory with this formula:

Memory (GB) ≈ (parameters in billions × bits per weight ÷ 8) + KV cache + ~10% overhead

At Q4_K_M quantization, weights average roughly 4.5-4.8 bits each. That puts a 70B model near 42 GB, a 32B model near 19 GB, and an 8B model near 5 GB.

The KV cache grows with context length. A long 32K-128K context can add several gigabytes to tens of gigabytes, depending on the model’s attention design. Always leave headroom.

2. Memory bandwidth: how fast will it generate?

Token generation (decode) is bandwidth-bound. Each new token requires reading the active weights from memory once. The ceiling is simple:

Max tokens/sec ≈ memory bandwidth (GB/s) ÷ active model size (GB)

A 42 GB model on a 256 GB/s Strix Halo tops out near 6 tok/s. The same model on a 1.2 TB/s M5 Ultra reaches the high 20s. Measured numbers from 2026 roundups match this math closely.

Prompt processing (prefill) is different. It is compute-bound, which is why NVIDIA hardware wins on long prompts with short answers. The DGX Spark processes prompts two to three times faster than a Strix Halo box despite near-identical generation speed.

Why Mixture-of-Experts (MoE) changes the math

MoE models activate only a fraction of their parameters per token. gpt-oss-120b holds about 117B parameters but activates roughly 5B per token. That lets a 128 GB, 256 GB/s machine deliver 34-56 tok/s on Strix Halo and 38-52 tok/s on DGX Spark, speeds that would be impossible for a dense 70B model on the same hardware.

Rule of thumb: big unified-memory boxes shine with MoE models and struggle with large dense models.

Discrete GPUs: Fastest Per Token, Smallest Memory

Discrete cards offer the highest bandwidth available to consumers. The RTX 5090 moves 1,792 GB/s against 546 GB/s for an M4 Max. If your model fits in 32 GB, nothing consumer-grade generates faster.

The catch is price. The RTX 5090 launched at $1,999, but Tom’s Hardware tracking put its best U.S. street price at $6,389 in September 2026, with third-party listings reaching $9,000+. The GDDR7 shortage is the driver, and reports suggest relief may not come until 2027-2028.

Best GPU options right now

Budget trick: two used RTX 3090s give you 48 GB of VRAM. llama.cpp splits layers across both cards for inference. Expect higher power draw, more heat, and a PSU of 1,000 W or more.

Unified-Memory Machines: Capacity First

These systems share one large memory pool between CPU and GPU. They load models no consumer GPU can hold.

Apple Silicon (Mac Mini, Mac Studio)

Apple offers the best combination of capacity, bandwidth, noise, and efficiency. The Mac Studio M4 Max (128 GB) delivers 546 GB/s, double the x86 unified-memory boxes at a similar street price of about $3,700. The M5 Ultra reaches up to 512 GB at roughly 1.2 TB/s, starting around $5,499 for the 96 GB base configuration.

Apple runs models through MLX and llama.cpp (Metal). Weak spots are long-prompt prefill speed and the lack of CUDA.

AMD Strix Halo (Ryzen AI Max+ 395)

The Ryzen AI Max+ 395 pairs 16 Zen 5 cores with a 40-CU Radeon 8060S and up to 128 GB LPDDR5X. Up to 96 GB can be allocated to the GPU, enough to hold a 70B Q4_K_M model (~42 GB) entirely on-GPU.

Bandwidth is about 256 GB/s. Expect ~4-6 tok/s on dense 70B and far better on MoE. Software has improved, and community testing currently favors llama.cpp with Vulkan over ROCm as the default. Set your BIOS memory allocation before first boot.

Street prices for 128 GB systems (Framework Desktop, GMKtec EVO-X2) sit around $3,449-$3,650, well above the roughly $2,000 launch-era pricing.

NVIDIA DGX Spark (GB10) and clones

The Spark carries 128 GB LPDDR5X at 273 GB/s with CUDA. Choose it when you need the NVIDIA toolchain: vLLM, TensorRT-LLM, NVFP4, or concurrent serving. The Founders Edition costs $4,699 after a February 2026 price hike. Partner GB10 systems like the ASUS Ascent GX10 have been listed lower, around $3,100, though stock varies. One published GX10 benchmark reached 60.6 tok/s on a gpt-oss-120b-class model.

Head-to-Head: 128 GB Class

Tokens/sec figures are derived from the bandwidth formula or community reports. Your results vary by runtime, context length, and quantization.

Hardware by Budget Tier

Setup: Run Your First Model

Option 1: Ollama (fastest start)

# Install, then pull and run a model
curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:120b   # needs ~64 GB+ memory

Option 2: llama.cpp server (full control, Strix Halo, NVIDIA, Mac)

# Build with the backend for your hardware (CUDA, Vulkan, or Metal)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_VULKAN=ON   # use -DGGML_CUDA=ON for NVIDIA
cmake --build build --config Release -j

# Serve a model with all layers offloaded and a 32K context
./build/bin/llama-server \
  -hf ggml-org/gpt-oss-120b-GGUF \
  -ngl 999 -c 32768 --jinja --port 8080

-ngl 999 offloads every layer to the GPU. -c 32768 sets the context window. Lower it if you run out of memory.

Option 3: MLX on Apple Silicon

pip install mlx-lm
mlx_lm.generate \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --prompt "Explain KV cache in two sentences." \
  --max-tokens 200

Call your local model like an API

llama-server and Ollama both expose an OpenAI-compatible endpoint:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="local")
resp = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Summarize the KV cache in 2 lines."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Real-World Workflows

Private coding assistant. Run a 30B-120B coding or reasoning model on a Strix Halo or Spark box and point your IDE at the local endpoint. Source code never leaves your network. MoE models keep generation speed usable.

Always-on home agent. A Mac Mini or Mac Studio idles at low wattage and stays silent. It suits 24/7 automation, summarization, and local RAG over personal documents.

Enterprise RAG without data egress. Pair a local embedding model with a vector database, then serve a mid-size model through vLLM on a CUDA machine. Concurrent serving is where the Spark and RTX PRO 6000 justify their price.

Long-document analysis. Large unified memory handles 64K-128K contexts that overflow a 24 GB card. Budget the KV cache first, then choose hardware.

Fine-tuning. LoRA on small models works on a Spark or Strix Halo. Serious training belongs on a 96 GB workstation card or the cloud.

Common Mistakes to Avoid

• Buying on VRAM alone. Bandwidth sets speed. A 128 GB box at 256 GB/s feels slow on dense 70B models.
• Ignoring context length. The model fits at 4K context and fails at 64K. Budget the KV cache.
• Skipping BIOS setup on Strix Halo. The default GPU allocation can be only 16-32 GB. Raise it before loading large models.
• Overpaying for an RTX 5090. At $5,000+, a Mac Studio M4 Max or two used 3090s often gives more usable capability.
• Expecting parity with frontier cloud models. Local models are strong but trail the largest hosted systems on hard reasoning tasks.

FAQ

How much VRAM do I need to run a 70B model locally?

Plan on ~42 GB for a 70B model at Q4_K_M, plus the KV cache and overhead, so 48-64 GB in practice. No single consumer GPU offers that. Use two 24 GB cards, a 96-128 GB unified-memory machine, or a 96 GB RTX PRO 6000.

Is a Mac or an NVIDIA GPU better for local LLMs?

If the model fits in 32 GB, an NVIDIA GPU is faster. For larger models, a Mac wins on capacity, bandwidth per dollar, noise, and power. Choose NVIDIA when you need CUDA tooling or fast prompt processing.

Is AMD Strix Halo good for local AI?

Yes, for capacity per dollar. It runs 70B-120B models in 128 GB at around 256 GB/s. It is best with MoE models, where 34-56 tok/s on gpt-oss-120b is reported. Dense 70B models run at about 4-6 tok/s, and prompt processing trails NVIDIA.

Should I buy now or wait for prices to drop?

The DRAM and GDDR7 shortage is raising prices across the board, and several reports point to 2027-2028 for relief. If you need hardware now, favor used GPUs and unified-memory systems. Buying only to save money on API fees rarely pays off at current prices.