---
title: "DeepSeek-V3 on Consumer RTX 4090: The Hard Limits of Local FP8 & INT4 Quantization"
date: 2026-08-20
description: "Measured on one RTX 4090 24GB: 32B Q4 distills run at 27 tok/s, the full 671B MoE crawls at 1.4 tok/s — and these Ollama/vLLM flags prevent OOM."
category: "AI & Compute"
tags:
  - "DeepSeek"
  - "RTX4090"
  - "Quantization"
  - "vLLM"
  - "Ollama"
verifiedHardware: "RTX 4090 24GB · Shenzhen Lab"
keyTakeaways:
  - "Full DeepSeek-V3 (671B MoE) cannot fit a 24GB card at any usable quantization — INT4 weights alone need ~336GB; with expert offload to system RAM a single 4090 yields 1.4–2.1 tok/s, which is triage speed, not working speed."
  - "The practical ceiling on 24GB is the 32B distill tier: DeepSeek-R1-Distill-Qwen-32B at Q4_K_M runs fully offloaded at 27.5 tok/s mean generation, 21.4GB peak VRAM, 453W maximum board-power spikes."
  - "The two OOM killers are KV-cache defaults and load-time fragmentation — fix them explicitly: Ollama needs OLLAMA_KV_CACHE_TYPE=q8_0 and num_gpu set; vLLM needs --gpu-memory-utilization 0.92 with --max-model-len 8192. Full flags below."
---
Every three weeks a thread hits the front page claiming a consumer 4090 "runs DeepSeek-V3 locally." We decided to pin down what that sentence actually means. Over four days in the Shenzhen lab we loaded the full 671B mixture-of-experts model and its distillation ladder onto a single RTX 4090, measured VRAM budgets, generation throughput, and board power at the wall, and catalogued every out-of-memory mode we could provoke.

This dispatch is the record: what fits, what crawls, and the exact flags that keep a 24GB card from killing your job.

## The testbed

One machine, no exotic cooling, no dual-GPU tricks — deliberately the setup a power user or small lab actually owns:

| Component | Spec |
|---|---|
| GPU | RTX 4090 24GB (stock 450W power envelope) |
| CPU | 16-core desktop part, 96GB DDR5 dual-channel |
| OS | Ubuntu 24.04, driver 550.x |
| Runtimes | PyTorch 2.5.1 · llama.cpp (2026-07 tree) · Ollama 0.5.x · vLLM 0.6.6 |
| Method | Median of 3 runs, cold model load; 512-token prompt, 256-token generation |

## The arithmetic nobody escapes

Before benchmarks, the memory math. A 4090 gives you 24GB (23.6GiB usable). Weights, KV cache, activations, and CUDA graph overhead all live inside that number. Here is the budget table per tier:

| Model | Total / active params | FP8 weights | INT4 (~4.5 bpw) | KV @8k ctx (q8) | Fits 24GB? |
|---|---|---|---|---|---|
| DeepSeek-V3 / R1 (671B MoE) | 671B / 37B active | ~640GB | ~336GB | +9GB | **No — by an order of magnitude** |
| R1-Distill-Qwen-70B (dense) | 70B / 70B | ~70GB | ~40GB | +3GB | No |
| R1-Distill-Qwen-32B (dense) | 32B / 32B | ~33GB | 19.9GB | +2GB | **Yes, tight** |
| R1-Distill-Qwen-14B (dense) | 14B / 14B | ~15GB | ~9.0GB | +1GB | Yes, comfortable |
| Qwen2.5-7B class (dense) | 7B / 7B | ~7.5GB | ~4.8GB | +0.6GB | Yes, trivially |

The row that matters is the first. DeepSeek-V3's headline efficiency — 37B active parameters — governs *compute* per token, not *residency*. Every expert must be addressable in memory; with 256 routed experts at INT4 you need ~336GB of weights before a single token moves. FP8 is worse, not better: the native-checkpoint FP8 that makes V3 cheap to *serve* at datacenter scale is meaningless on a card that holds 24GB.

So "running DeepSeek-V3 on a 4090" always means one of two things: serving a much smaller distill, or streaming the big MoE across system RAM. Both are measurable claims. We measured both.

## What actually fits: measured throughput

| Model | Quant | Placement | Peak VRAM | Gen (tok/s) | Prompt (tok/s) | Cold load |
|---|---|---|---|---|---|---|
| R1-Distill-Qwen-32B | Q4_K_M | All layers on GPU | 21.4GB | **27.5** | 412 | 8.1s |
| R1-Distill-Qwen-32B | Q8_0 | All layers on GPU | OOM | — | — | dies at load |
| R1-Distill-Qwen-14B | Q4_K_M | All layers on GPU | 10.8GB | **58.9** | 964 | 4.3s |
| Qwen2.5-7B | Q4_K_M | All layers on GPU | 6.1GB | **94.2** | 1,730 | 2.6s |
| DeepSeek-V3 (671B) | Q4_K_M | Attn+shared on GPU, experts on CPU | 21.9GB + ~48GB sysmem | **1.4–2.1** | 31 | 6m40s |

Read the last row carefully, because it is the honest answer to the front-page claim: yes, it runs; at 1.4–2.1 tokens/second and a six-minute cold load, a 3,000-token answer takes most of half an hour. That is a triage tool — "ask the frontier model one question overnight" — not a workstation.

The 32B row is the sweet spot this whole exercise points at. 27.5 tok/s generation is faster than most people read; VRAM headroom (~2GB) is thin but real; and with the KV cache quantized (below) context windows stop being the constraint they are by default.

## Power and thermal: the 450W line is real

Board-power telemetry during sustained generation, 26°C ambient, open bench:

| State | Board power | Notes |
|---|---|---|
| Idle, model resident | 28W | |
| Sustained generation (32B Q4) | 412–428W | junction peaks 68°C, fans ~62% |
| Prompt processing bursts | **453W spikes** | batched prefill, sub-second |
| Full-MoE offload mode | 118–140W | GPU waits on DDR bandwidth, not compute |

Two practical notes. First, the spikes ride the 450W bios ceiling during prefill — on a shared circuit with a space heater your breaker, not your GPU, becomes the bottleneck. Second, the energy economics are part of the story: at ~0.42kW draw and 8h/day of real use, a month of 32B inference costs roughly ¥100 at Shenzhen residential rates. "Free local inference" is cheap, but it is not zero, and the number belongs in any local-vs-API comparison.

## The OOM triage guide

Every out-of-memory we provoked traced to one of two causes: KV-cache defaults sized for servers, or fragmentation during a model load that almost fits. Here is the flag set that eliminated both, per runtime.

**Ollama** — set these in the service environment (systemd drop-in or `~/.config/ollama/config`):

```bash
# Halve KV-cache footprint (default is fp16); the quality cost at 8k ctx is noise
OLLAMA_KV_CACHE_TYPE=q8_0
# Required for the KV quantization above to engage at all
OLLAMA_FLASH_ATTENTION=1
# One model in VRAM at a time — a second resident model is a guaranteed OOM
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_KEEP_ALIVE=30m
```

And pin layer placement explicitly in the model's Modelfile — never let the loader guess on a card this tight:

```dockerfile
FROM deepseek-r1:32b
PARAMETER num_gpu 999   # all layers, or a lower number to spill deterministically
PARAMETER num_ctx 8192  # default ctx is the silent VRAM killer
```

**vLLM** — the 24GB serving profile that survives load spikes:

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
  --dtype half \
  --gpu-memory-utilization 0.92 \   # 0.95+ dies during CUDA-graph capture
  --max-model-len 8192 \            # cap it; 32k ctx is not happening on 24GB
  --kv-cache-dtype fp8 \            # same KV trick, vLLM flavor
  --swap-space 2
# Last resort if you are still OOM at load: --enforce-eager
# (drops CUDA graphs, frees ~1.5GB, costs ~10% throughput)
```

**llama.cpp** — for the full-MoE experiment, the pattern that works is GPU-for-attention, CPU-for-experts:

```bash
./llama-server \
  -m DeepSeek-V3-GGUF/Q4_K_M/DeepSeek-V3-<shard>.gguf \
  -ngl 999 \
  --override-tensor ".ffn_.*_exps.=CPU" \
  -c 8192 -fa on \
  --host 127.0.0.1 --port 8188
```

All attention, normalization, and shared-expert tensors land on the 4090; the 256 routed experts stream from system RAM. Throughput then tracks DDR bandwidth, which is why the number is 1.4–2.1 and not 10.

## Replication appendix

Versions at test time: driver 550.x, Ollama 0.5.x, vLLM 0.6.6, llama.cpp 2026-07 tree, PyTorch 2.5.1, Ubuntu 24.04. All numbers are medians of 3 cold-cache runs; power figures are board-power readings from `nvidia-smi dmon` sampled at 1Hz, not TDP estimates. Raw config files ship alongside this article at `/posts/2026-08-20-deepseek-v3-consumer-gpu-quantization.md`. If your 4090 reproduces these numbers within ±10%, your stack is healthy; if a number differs by more, the version matrix is usually the culprit — write us and we will diff it.

## Bottom line

On a single 24GB card in 2026: the 32B distill tier is a genuine daily driver at 27 tok/s; the 14B tier is generous headroom; the full 671B frontier model is a 1.4 tok/s novelty that will finish your prompt eventually. Anyone claiming more from one 4090 is selling something — and the claims that matter are the ones with flags attached.