19 Nov 2026 · gpu-sizing · vllm · kv-cache · l40s
A 32B model on a 48 GB GPU: what actually fits
Qwen3-32B on one NVIDIA L40S: why full precision fails, what 8-bit buys, and where the context length really runs out.
“32 billion parameters, 48 GB of GPU memory, should be fine.” It isn’t, and the reason is worth two minutes, because the same arithmetic decides every self-hosting budget.
These are estimates from gpu-fit, using the model’s published settings and vLLM’s memory rules. A measured follow-up on real hardware is coming; I’ll update this post with the differences.
The budget
An L40S has 48 GB, which is about 45 GiB usable. vLLM takes 90% of that by default
(gpu_memory_utilization 0.9): 40.5 GiB. Out of that come three things:
- Weights, fixed by the model and its precision.
- Runtime overhead: CUDA graphs, activations, buffers. About 2 GiB assumed here.
- KV cache: whatever is left. This is what decides how long a request can be and how many requests run at once.
Full precision: does not fit at all
| Qwen3-32B (BF16) | |
|---|---|
| Weights | 61.0 GiB |
| Budget | 40.5 GiB |
| Left for KV cache | −22.5 GiB |
At 2 bytes per parameter the weights alone are bigger than the whole budget. No setting fixes that; you need two GPUs or fewer bits.
8-bit weights: fits, with a catch
| Qwen3-32B-FP8 | |
|---|---|
| Weights | 32.0 GiB |
| Runtime overhead | 2.0 GiB |
| Left for KV cache | 6.5 GiB |
| KV cache capacity | 26,737 tokens |
Each token of context costs this model about 256 KiB of KV cache (64 layers × 8 KV heads × 128 dims × 2 for keys and values × 2 bytes). 6.5 GiB ÷ 256 KiB ≈ 26,700 tokens, shared by every request in flight.
| Context you ask for | Requests that fit at full length |
|---|---|
| 8K | 3.3 |
| 32K | 0.8: one 32K request does not fit |
| 128K | 0.2 |
So “it fits” is true only up to about 27K tokens, for one user.
8-bit KV cache too: a real 32K
Storing the KV cache in FP8 (--kv-cache-dtype fp8) halves its cost per token:
| FP8 weights + FP8 KV cache | |
|---|---|
| KV cache capacity | 53,475 tokens |
| Requests at 32K | 1.6 |
| Longest single request | 40,960 tokens, the model’s own limit |
Two things to notice:
- The longest request is now capped by the model (its 40,960-token maximum), not by memory.
- FP8 KV cache costs some accuracy. Whether that matters depends on the task; measure it on your own prompts before relying on it.
What to take away
- Parameters × bytes per parameter tells you if the model loads. It tells you nothing about how much context you get.
- The KV cache is the real budget, and every concurrent request shares it.
- On one 48 GB GPU, a 32B model is a single-user, roughly 27K-token setup at 8-bit, or about 50K tokens shared with an 8-bit KV cache. More than that needs a bigger GPU or two of them.
Check your own model and GPU:
pip install gpu-fit
gpu-fit Qwen/Qwen3-32B-FP8 --gpu L40S --context 32k --kv-cache-dtype fp8
Estimates, not measurements. Updated when the measured run is done.