Guide4 sections2 min read
Estimate memory before comparing speed
Begin with the model weights, precision, context length, batch size and serving framework. Memory is also needed for the KV cache and runtime overhead.
As a rough weight-only calculation, 8 billion parameters at 2 bytes per parameter require about 16 GB in decimal units. This excludes runtime overhead and is not a guarantee that a model will fit on a particular GPU.
Define the result that matters
Interactive applications care about response latency and time to first token. Batch workloads may care more about throughput. Set a representative prompt length, output length and concurrency before testing.
GPU specification tables help narrow a shortlist. They do not replace a benchmark of your actual model and software configuration.
Check the full deployment
Confirm framework support, drivers, available storage, model loading time and network requirements. If multiple GPUs are needed, verify the actual interconnect and framework support on the chosen offer.
Choose a region based on the available offer and your workload requirements. Do not infer data location or contractual protections from the marketplace brand alone.
Compare cost per useful result
Run the same workload on each candidate, recording the configuration, software versions, rental price, runtime and failed requests. Compare total session cost as well as hourly price.
Record the test date and limits. A result from one prompt mix or concurrency level may not generalize to another application.
Estimate model weight memory
Start with the number of model parameters and weight precision. This calculates weight storage only, not the total GPU memory required to run a model.
Weight storage only: 16 GB (decimal)
Formula: billion parameters × bits per weight ÷ 8 = decimal GB. Add KV cache, activations, runtime allocations and any quantization metadata separately. Actual requirements depend on the architecture, context, batch size and software. A weight-only result does not establish that a model fits on a GPU, or that a precision is supported.
Read the Transformers quantization overview