Projects
Things I’ve built in the open
gpu-fit
PythonvLLMMITv0.2
How many users can this LLM serve on this GPU, and with what command?
Most calculators multiply parameters by two and add a guess. Modern models break that.
gpu-fit reads the model’s real architecture from Hugging Face and mirrors vLLM’s memory budget:
weights, runtime overhead, KV cache, requests at full context and the longest single request.
Then it prints the vllm serve command.
- Understands grouped-query attention, hybrid linear-attention models, DeepSeek MLA, sliding windows, FP8/AWQ/GPTQ checkpoints, tensor parallelism and FP8 KV caches.
- Validated against real vLLM runs on an NVIDIA L40S.
- Warns you before vLLM refuses to start a hybrid model with too many request slots.
- No dependencies.
pip install gpu-fit
gpu-fit Qwen/Qwen3-32B --context 128k --concurrency 4
vllm-ops-kit coming next
A production setup for a single GPU server: systemd services, model staging to local NVMe so restarts are fast, health checks, an API key in front, and HTTPS. The things most “docker run” guides leave out.