Skip to content

Benchmark vLLM Inference on a Cloud GPU

Prepare a local vLLM benchmark with fixed model, token lengths and concurrency. Record latency, failures and full session cost.

GuidePublished by EverXCloud Team4 sections2 min read

ShareXLinkedInReddit

Fix the Model and Memory Budget

Select a text-generation model supported by your installed vLLM release, with a pinned revision and license that permits your use. Keep the model and tokenizer in a local directory on an authorized test machine. The commands below assume vLLM is already installed; they do not provision an EverXCloud GPU or download model weights.

Record weight precision, quantization, maximum context, GPU count and concurrency. Account for weights, KV cache and runtime memory. An 8B model has about 16 GB of BF16 weights in decimal units, before other allocations. This arithmetic alone does not establish that a model fits or that a GPU supports the chosen format.

Start a Local Test Server

Replace the example directory with your existing local model directory. Check vllm --version and vllm serve --help first; options and supported architectures depend on the release. The 2,048-token context and single GPU are test settings, not a capacity recommendation.

Keep the server bound to localhost. In a second terminal on the same test machine, wait for startup to complete, then check /v1/models. Record model loading time separately. Do not expose the unauthenticated test endpoint as a public service.

Start vLLM with an existing local model
vllm --version
vllm serve /absolute/path/to/local-model \
  --host 127.0.0.1 --port 8000 \
  --served-model-name benchmark-model \
  --max-model-len 2048 --tensor-parallel-size 1

Run a Bounded Synthetic Workload

Check the benchmark command's help against your installed release. Use the same tokenizer directory as the server. The example sends 20 synthetic requests with a 256-token input target, a 64-token output target and at most one concurrent request. Random prompts test a load shape; they do not evaluate answer quality or represent your production traffic.

Run a warm-up separately, then repeat the recorded test without changing the model, cache settings or prompt lengths. Record prefix-cache and warm-up conditions. Follow with representative, non-sensitive application prompts and a concurrency level that reflects your workload.

Run a small local vLLM serving benchmark
curl --fail http://127.0.0.1:8000/v1/models
vllm bench serve \
  --backend openai --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions --model benchmark-model \
  --tokenizer /absolute/path/to/local-model \
  --dataset-name random --random-input-len 256 \
  --random-output-len 64 --num-prompts 20 \
  --max-concurrency 1 --seed 0

Compare Completed Work, Latency, and Cost

Retain the benchmark output and server logs. Record completed and failed requests, actual input and output tokens, time to first token, latency percentiles and output-token throughput. State whether a measurement includes queueing and model loading. Do not derive a speed multiplier from manufacturer memory bandwidth alone.

Compare total session cost per successful output, including setup and failed runs. Synthetic results do not establish production latency. Stop the test server and release rented compute when finished, after saving results and checking persistent storage. No EverXCloud benchmark result is claimed by this preparation guide.

Sources

Keep Reading

All guides

Next guide2 min read

Compare Cloud GPUs with a ComfyUI Workflow

Keep a ComfyUI workflow reproducible across GPUs: checkpoint, nodes, resolution, seeds, uncached runs and cost per successful image.

Guide 213 min read

GPU Passive Income: What to Expect

What really determines what a rented GPU earns: occupancy, listed price, commission and running costs, with a method to estimate your own net income.

Request capacity

Get EverXCloud launch updates

New GPU capacity, regions, and marketplace features by email. Double opt-in, unsubscribe at any time.

Subscribe