DeepSeek-V3 on Consumer RTX 4090: The Hard Limits of Local FP8 & INT4 Quantization

Every three weeks a thread hits the front page claiming a consumer 4090 “runs DeepSeek-V3 locally.” We decided to pin down what that sentence actually means. Over four days in the Shenzhen lab we loaded the full 671B mixture-of-experts model and its distillation ladder onto a single RTX 4090, measured VRAM budgets, generation throughput, and board power at the wall, and catalogued every out-of-memory mode we could provoke.

This dispatch is the record: what fits, what crawls, and the exact flags that keep a 24GB card from killing your job.

The testbed

One machine, no exotic cooling, no dual-GPU tricks — deliberately the setup a power user or small lab actually owns:

Component	Spec
GPU	RTX 4090 24GB (stock 450W power envelope)
CPU	16-core desktop part, 96GB DDR5 dual-channel
OS	Ubuntu 24.04, driver 550.x
Runtimes	PyTorch 2.5.1 · llama.cpp (2026-07 tree) · Ollama 0.5.x · vLLM 0.6.6
Method	Median of 3 runs, cold model load; 512-token prompt, 256-token generation

The arithmetic nobody escapes

Before benchmarks, the memory math. A 4090 gives you 24GB (23.6GiB usable). Weights, KV cache, activations, and CUDA graph overhead all live inside that number. Here is the budget table per tier:

Model	Total / active params	FP8 weights	INT4 (~4.5 bpw)	KV @8k ctx (q8)	Fits 24GB?
DeepSeek-V3 / R1 (671B MoE)	671B / 37B active	~640GB	~336GB	+9GB	No — by an order of magnitude
R1-Distill-Qwen-70B (dense)	70B / 70B	~70GB	~40GB	+3GB	No
R1-Distill-Qwen-32B (dense)	32B / 32B	~33GB	19.9GB	+2GB	Yes, tight
R1-Distill-Qwen-14B (dense)	14B / 14B	~15GB	~9.0GB	+1GB	Yes, comfortable
Qwen2.5-7B class (dense)	7B / 7B	~7.5GB	~4.8GB	+0.6GB	Yes, trivially

The row that matters is the first. DeepSeek-V3’s headline efficiency — 37B active parameters — governs compute per token, not residency. Every expert must be addressable in memory; with 256 routed experts at INT4 you need ~336GB of weights before a single token moves. FP8 is worse, not better: the native-checkpoint FP8 that makes V3 cheap to serve at datacenter scale is meaningless on a card that holds 24GB.

So “running DeepSeek-V3 on a 4090” always means one of two things: serving a much smaller distill, or streaming the big MoE across system RAM. Both are measurable claims. We measured both.

What actually fits: measured throughput

Model	Quant	Placement	Peak VRAM	Gen (tok/s)	Prompt (tok/s)	Cold load
R1-Distill-Qwen-32B	Q4_K_M	All layers on GPU	21.4GB	27.5	412	8.1s
R1-Distill-Qwen-32B	Q8_0	All layers on GPU	OOM	—	—	dies at load
R1-Distill-Qwen-14B	Q4_K_M	All layers on GPU	10.8GB	58.9	964	4.3s
Qwen2.5-7B	Q4_K_M	All layers on GPU	6.1GB	94.2	1,730	2.6s
DeepSeek-V3 (671B)	Q4_K_M	Attn+shared on GPU, experts on CPU	21.9GB + ~48GB sysmem	1.4–2.1	31	6m40s

Read the last row carefully, because it is the honest answer to the front-page claim: yes, it runs; at 1.4–2.1 tokens/second and a six-minute cold load, a 3,000-token answer takes most of half an hour. That is a triage tool — “ask the frontier model one question overnight” — not a workstation.

The 32B row is the sweet spot this whole exercise points at. 27.5 tok/s generation is faster than most people read; VRAM headroom (~2GB) is thin but real; and with the KV cache quantized (below) context windows stop being the constraint they are by default.

Power and thermal: the 450W line is real

Board-power telemetry during sustained generation, 26°C ambient, open bench:

State	Board power	Notes
Idle, model resident	28W
Sustained generation (32B Q4)	412–428W	junction peaks 68°C, fans ~62%
Prompt processing bursts	453W spikes	batched prefill, sub-second
Full-MoE offload mode	118–140W	GPU waits on DDR bandwidth, not compute

Two practical notes. First, the spikes ride the 450W bios ceiling during prefill — on a shared circuit with a space heater your breaker, not your GPU, becomes the bottleneck. Second, the energy economics are part of the story: at ~0.42kW draw and 8h/day of real use, a month of 32B inference costs roughly ¥100 at Shenzhen residential rates. “Free local inference” is cheap, but it is not zero, and the number belongs in any local-vs-API comparison.

The OOM triage guide

Every out-of-memory we provoked traced to one of two causes: KV-cache defaults sized for servers, or fragmentation during a model load that almost fits. Here is the flag set that eliminated both, per runtime.

Ollama — set these in the service environment (systemd drop-in or ~/.config/ollama/config):

# Halve KV-cache footprint (default is fp16); the quality cost at 8k ctx is noise
OLLAMA_KV_CACHE_TYPE=q8_0
# Required for the KV quantization above to engage at all
OLLAMA_FLASH_ATTENTION=1
# One model in VRAM at a time — a second resident model is a guaranteed OOM
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_KEEP_ALIVE=30m

And pin layer placement explicitly in the model’s Modelfile — never let the loader guess on a card this tight:

FROM deepseek-r1:32b
PARAMETER num_gpu 999   # all layers, or a lower number to spill deterministically
PARAMETER num_ctx 8192  # default ctx is the silent VRAM killer

vLLM — the 24GB serving profile that survives load spikes:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --dtype half \
  --gpu-memory-utilization 0.92 \   # 0.95+ dies during CUDA-graph capture
  --max-model-len 8192 \            # cap it; 32k ctx is not happening on 24GB
  --kv-cache-dtype fp8 \            # same KV trick, vLLM flavor
  --swap-space 2
# Last resort if you are still OOM at load: --enforce-eager
# (drops CUDA graphs, frees ~1.5GB, costs ~10% throughput)

llama.cpp — for the full-MoE experiment, the pattern that works is GPU-for-attention, CPU-for-experts:

./llama-server \
  -m DeepSeek-V3-GGUF/Q4_K_M/DeepSeek-V3-<shard>.gguf \
  -ngl 999 \
  --override-tensor ".ffn_.*_exps.=CPU" \
  -c 8192 -fa on \
  --host 127.0.0.1 --port 8188

All attention, normalization, and shared-expert tensors land on the 4090; the 256 routed experts stream from system RAM. Throughput then tracks DDR bandwidth, which is why the number is 1.4–2.1 and not 10.

Replication appendix

Versions at test time: driver 550.x, Ollama 0.5.x, vLLM 0.6.6, llama.cpp 2026-07 tree, PyTorch 2.5.1, Ubuntu 24.04. All numbers are medians of 3 cold-cache runs; power figures are board-power readings from nvidia-smi dmon sampled at 1Hz, not TDP estimates. Raw config files ship alongside this article at /posts/2026-08-20-deepseek-v3-consumer-gpu-quantization.md. If your 4090 reproduces these numbers within ±10%, your stack is healthy; if a number differs by more, the version matrix is usually the culprit — write us and we will diff it.

Bottom line

On a single 24GB card in 2026: the 32B distill tier is a genuine daily driver at 27 tok/s; the 14B tier is generous headroom; the full 671B frontier model is a 1.4 tok/s novelty that will finish your prompt eventually. Anyone claiming more from one 4090 is selling something — and the claims that matter are the ones with flags attached.