Projects

Things I’ve built in the open

gpu-fit

PythonvLLMMITv0.2

How many users can this LLM serve on this GPU, and with what command?

Most calculators multiply parameters by two and add a guess. Modern models break that. gpu-fit reads the model’s real architecture from Hugging Face and mirrors vLLM’s memory budget: weights, runtime overhead, KV cache, requests at full context and the longest single request. Then it prints the vllm serve command.

  • Understands grouped-query attention, hybrid linear-attention models, DeepSeek MLA, sliding windows, FP8/AWQ/GPTQ checkpoints, tensor parallelism and FP8 KV caches.
  • Validated against real vLLM runs on an NVIDIA L40S.
  • Warns you before vLLM refuses to start a hybrid model with too many request slots.
  • No dependencies.
pip install gpu-fit
gpu-fit Qwen/Qwen3-32B --context 128k --concurrency 4

GitHub · PyPI

vllm-ops-kit coming next

A production setup for a single GPU server: systemd services, model staging to local NVMe so restarts are fast, health checks, an API key in front, and HTTPS. The things most “docker run” guides leave out.