GuidePublished by EverXCloud Team4 sections2 min read
Fix the Model and Memory Budget
Select a text-generation model supported by your installed vLLM release, with a pinned revision and license that permits your use. Keep the model and tokenizer in a local directory on an authorized test machine. The commands below assume vLLM is already installed; they do not provision an EverXCloud GPU or download model weights.
Record weight precision, quantization, maximum context, GPU count and concurrency. Account for weights, KV cache and runtime memory. An 8B model has about 16 GB of BF16 weights in decimal units, before other allocations. This arithmetic alone does not establish that a model fits or that a GPU supports the chosen format.
Start a Local Test Server
Replace the example directory with your existing local model directory. Check vllm --version and vllm serve --help first; options and supported architectures depend on the release. The 2,048-token context and single GPU are test settings, not a capacity recommendation.
Keep the server bound to localhost. In a second terminal on the same test machine, wait for startup to complete, then check /v1/models. Record model loading time separately. Do not expose the unauthenticated test endpoint as a public service.
vllm --version
vllm serve /absolute/path/to/local-model \
--host 127.0.0.1 --port 8000 \
--served-model-name benchmark-model \
--max-model-len 2048 --tensor-parallel-size 1Run a Bounded Synthetic Workload
Check the benchmark command's help against your installed release. Use the same tokenizer directory as the server. The example sends 20 synthetic requests with a 256-token input target, a 64-token output target and at most one concurrent request. Random prompts test a load shape; they do not evaluate answer quality or represent your production traffic.
Run a warm-up separately, then repeat the recorded test without changing the model, cache settings or prompt lengths. Record prefix-cache and warm-up conditions. Follow with representative, non-sensitive application prompts and a concurrency level that reflects your workload.
curl --fail http://127.0.0.1:8000/v1/models
vllm bench serve \
--backend openai --base-url http://127.0.0.1:8000 \
--endpoint /v1/completions --model benchmark-model \
--tokenizer /absolute/path/to/local-model \
--dataset-name random --random-input-len 256 \
--random-output-len 64 --num-prompts 20 \
--max-concurrency 1 --seed 0Compare Completed Work, Latency, and Cost
Retain the benchmark output and server logs. Record completed and failed requests, actual input and output tokens, time to first token, latency percentiles and output-token throughput. State whether a measurement includes queueing and model loading. Do not derive a speed multiplier from manufacturer memory bandwidth alone.
Compare total session cost per successful output, including setup and failed runs. Synthetic results do not establish production latency. Stop the test server and release rented compute when finished, after saving results and checking persistent storage. No EverXCloud benchmark result is claimed by this preparation guide.
Sources
- vLLM serving command reference (checked October 28, 2026)
- vLLM serving benchmark reference (checked October 28, 2026)
