<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="http://0.0.0.0:4000/feed.xml" rel="self" type="application/atom+xml" /><link href="http://0.0.0.0:4000/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-19T15:18:25+00:00</updated><id>http://0.0.0.0:4000/feed.xml</id><title type="html">Kiran Tallapragada</title><subtitle>I build AI and data platforms for regulated industries: life sciences, tax &amp; finance and digital assets. Self-hosted LLMs, document AI and the cloud underneath, with real measured numbers.</subtitle><author><name>Kiran Tallapragada</name></author><entry><title type="html">A 32B model on a 48 GB GPU: what actually fits</title><link href="http://0.0.0.0:4000/2026/11/a-32b-model-on-a-48gb-gpu/" rel="alternate" type="text/html" title="A 32B model on a 48 GB GPU: what actually fits" /><published>2026-11-19T12:34:12+00:00</published><updated>2026-11-19T12:34:12+00:00</updated><id>http://0.0.0.0:4000/2026/11/a-32b-model-on-a-48gb-gpu</id><content type="html" xml:base="http://0.0.0.0:4000/2026/11/a-32b-model-on-a-48gb-gpu/"><![CDATA[<p>“32 billion parameters, 48 GB of GPU memory, should be fine.” It isn’t, and the reason is
worth two minutes, because the same arithmetic decides every self-hosting budget.</p>

<p>These are <strong>estimates</strong> from <a href="https://github.com/KiranTallapragada/gpu-fit">gpu-fit</a>, using
the model’s published settings and vLLM’s memory rules. A measured follow-up on real
hardware is coming; I’ll update this post with the differences.</p>

<h2 id="the-budget">The budget</h2>

<p>An L40S has 48 GB, which is about 45 GiB usable. vLLM takes 90% of that by default
(<code>gpu_memory_utilization 0.9</code>): <strong>40.5 GiB</strong>. Out of that come three things:</p>

<ol>
  <li><strong>Weights</strong>, fixed by the model and its precision.</li>
  <li><strong>Runtime overhead</strong>: CUDA graphs, activations, buffers. About 2 GiB assumed here.</li>
  <li><strong>KV cache</strong>: whatever is left. This is what decides how long a request can be and how
many requests run at once.</li>
</ol>

<h2 id="full-precision-does-not-fit-at-all">Full precision: does not fit at all</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Qwen3-32B (BF16)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Weights</td>
      <td>61.0 GiB</td>
    </tr>
    <tr>
      <td>Budget</td>
      <td>40.5 GiB</td>
    </tr>
    <tr>
      <td>Left for KV cache</td>
      <td><strong>−22.5 GiB</strong></td>
    </tr>
  </tbody>
</table>

<p>At 2 bytes per parameter the weights alone are bigger than the whole budget. No setting
fixes that; you need two GPUs or fewer bits.</p>

<h2 id="8-bit-weights-fits-with-a-catch">8-bit weights: fits, with a catch</h2>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>Qwen3-32B-FP8</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Weights</td>
      <td>32.0 GiB</td>
    </tr>
    <tr>
      <td>Runtime overhead</td>
      <td>2.0 GiB</td>
    </tr>
    <tr>
      <td>Left for KV cache</td>
      <td>6.5 GiB</td>
    </tr>
    <tr>
      <td>KV cache capacity</td>
      <td><strong>26,737 tokens</strong></td>
    </tr>
  </tbody>
</table>

<p>Each token of context costs this model about <strong>256 KiB</strong> of KV cache
(64 layers × 8 KV heads × 128 dims × 2 for keys and values × 2 bytes).
6.5 GiB ÷ 256 KiB ≈ 26,700 tokens, <strong>shared by every request in flight</strong>.</p>

<table>
  <thead>
    <tr>
      <th>Context you ask for</th>
      <th>Requests that fit at full length</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>8K</td>
      <td>3.3</td>
    </tr>
    <tr>
      <td>32K</td>
      <td><strong>0.8</strong>: one 32K request does not fit</td>
    </tr>
    <tr>
      <td>128K</td>
      <td>0.2</td>
    </tr>
  </tbody>
</table>

<p>So “it fits” is true only up to about 27K tokens, for one user.</p>

<h2 id="8-bit-kv-cache-too-a-real-32k">8-bit KV cache too: a real 32K</h2>

<p>Storing the KV cache in FP8 (<code>--kv-cache-dtype fp8</code>) halves its cost per token:</p>

<table>
  <thead>
    <tr>
      <th> </th>
      <th>FP8 weights + FP8 KV cache</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>KV cache capacity</td>
      <td><strong>53,475 tokens</strong></td>
    </tr>
    <tr>
      <td>Requests at 32K</td>
      <td>1.6</td>
    </tr>
    <tr>
      <td>Longest single request</td>
      <td>40,960 tokens, the model’s own limit</td>
    </tr>
  </tbody>
</table>

<p>Two things to notice:</p>
<ul>
  <li>The longest request is now capped by the <strong>model</strong> (its 40,960-token maximum), not by memory.</li>
  <li>FP8 KV cache costs some accuracy. Whether that matters depends on the task; measure it on
your own prompts before relying on it.</li>
</ul>

<h2 id="what-to-take-away">What to take away</h2>

<ul>
  <li>Parameters × bytes per parameter tells you if the model loads. It tells you nothing about
how much context you get.</li>
  <li>The KV cache is the real budget, and every concurrent request shares it.</li>
  <li>On one 48 GB GPU, a 32B model is a single-user, roughly 27K-token setup at 8-bit, or about
50K tokens shared with an 8-bit KV cache. More than that needs a bigger GPU or two of them.</li>
</ul>

<p>Check your own model and GPU:</p>

<pre><code>pip install gpu-fit
gpu-fit Qwen/Qwen3-32B-FP8 --gpu L40S --context 32k --kv-cache-dtype fp8
</code></pre>

<p><em>Estimates, not measurements. Updated when the measured run is done.</em></p>]]></content><author><name>Kiran Tallapragada</name></author><category term="gpu-sizing" /><category term="vllm" /><category term="kv-cache" /><category term="l40s" /><summary type="html"><![CDATA[Qwen3-32B on one NVIDIA L40S: why full precision fails, what 8-bit buys, and where the context length really runs out.]]></summary></entry></feed>