19 Nov 2026 · gpu-sizing · vllm · kv-cache · l40s

A 32B model on a 48 GB GPU: what actually fits

Qwen3-32B on one NVIDIA L40S: why full precision fails, what 8-bit buys, and where the context length really runs out.

“32 billion parameters, 48 GB of GPU memory, should be fine.” It isn’t, and the reason is worth two minutes, because the same arithmetic decides every self-hosting budget.

These are estimates from gpu-fit, using the model’s published settings and vLLM’s memory rules. A measured follow-up on real hardware is coming; I’ll update this post with the differences.

The budget

An L40S has 48 GB, which is about 45 GiB usable. vLLM takes 90% of that by default (gpu_memory_utilization 0.9): 40.5 GiB. Out of that come three things:

  1. Weights, fixed by the model and its precision.
  2. Runtime overhead: CUDA graphs, activations, buffers. About 2 GiB assumed here.
  3. KV cache: whatever is left. This is what decides how long a request can be and how many requests run at once.

Full precision: does not fit at all

  Qwen3-32B (BF16)
Weights 61.0 GiB
Budget 40.5 GiB
Left for KV cache −22.5 GiB

At 2 bytes per parameter the weights alone are bigger than the whole budget. No setting fixes that; you need two GPUs or fewer bits.

8-bit weights: fits, with a catch

  Qwen3-32B-FP8
Weights 32.0 GiB
Runtime overhead 2.0 GiB
Left for KV cache 6.5 GiB
KV cache capacity 26,737 tokens

Each token of context costs this model about 256 KiB of KV cache (64 layers × 8 KV heads × 128 dims × 2 for keys and values × 2 bytes). 6.5 GiB ÷ 256 KiB ≈ 26,700 tokens, shared by every request in flight.

Context you ask for Requests that fit at full length
8K 3.3
32K 0.8: one 32K request does not fit
128K 0.2

So “it fits” is true only up to about 27K tokens, for one user.

8-bit KV cache too: a real 32K

Storing the KV cache in FP8 (--kv-cache-dtype fp8) halves its cost per token:

  FP8 weights + FP8 KV cache
KV cache capacity 53,475 tokens
Requests at 32K 1.6
Longest single request 40,960 tokens, the model’s own limit

Two things to notice:

What to take away

Check your own model and GPU:

pip install gpu-fit
gpu-fit Qwen/Qwen3-32B-FP8 --gpu L40S --context 32k --kv-cache-dtype fp8

Estimates, not measurements. Updated when the measured run is done.

← All notes