> ## Documentation Index > Fetch the complete documentation index at: https://docs.warpscale.ai/llms.txt > Use this file to discover all available pages before exploring further. # Set up vLLM > Point the probe at vLLM, start the server with the plugin enabled, and confirm telemetry arrives. ## Before you start * A GPU with the NVIDIA driver installed. For the Docker path, also the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). * The [probe](/install/probe), running on the same node * An **organization API key**. Create one under **API keys** in the dashboard. These examples assume vLLM serves on its default `127.0.0.1:8000`. Adjust the URLs if yours differs. ## 1. Start the probe with vLLM settings The plugin reports what happens inside the engine. These two settings add the server's own metrics and traces. ```bash theme={null} sudo warpscaled \ --api-address :443 \ --api-token \ --vllm-metrics-url http://127.0.0.1:8000/metrics \ --enable-otlp-receiver ``` `--vllm-metrics-url` points at vLLM's Prometheus endpoint. `--enable-otlp-receiver` starts a receiver on `127.0.0.1:4317`. vLLM exports per-request traces to it in the next step. Start the probe first — it polls on `--vllm-poll-interval` until vLLM comes up. See the [full flag list](/install/probe#configuration). ## 2. Start vLLM The plugin runs inside vLLM and the probe runs beside it, so it needs to know where to send what it collects. The plugin has to live in the same Python environment as vLLM. vLLM's official image already carries a matched torch, CUDA, and NCCL. Add the plugin on start. ```bash theme={null} docker run --rm --runtime nvidia --gpus all --ipc=host --network host \ -v /var/run/warpscale:/var/run/warpscale \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e WS_VLLM_SINK=unix:///var/run/warpscale/warpscaled.sock \ -e OTEL_EXPORTER_OTLP_TRACES_INSECURE=true \ --entrypoint bash vllm/vllm-openai:v0.23.0 -c ' pip install -q warpscale-vllm exec vllm serve Qwen/Qwen2.5-0.5B-Instruct \ --distributed-executor-backend warpscale_vllm.executor.WarpscaleUniExecutor \ --middleware warpscale_vllm.middleware.TenantCapture \ --otlp-traces-endpoint http://127.0.0.1:4317 ' ``` `--network host` is required, because the probe runs on the host and both connections between the two use the host's loopback. The probe scrapes vLLM at `127.0.0.1:8000`, and vLLM sends traces to `127.0.0.1:4317`. A container with its own network reaches neither. `-v /var/run/warpscale:/var/run/warpscale` is required. The probe's socket is a file, so sharing the network namespace does not reach it. Without the mount the plugin has nowhere to send records and vLLM starts normally with no error. For anything long-lived, bake the plugin into an image instead of installing it on every start. ```dockerfile Dockerfile theme={null} FROM vllm/vllm-openai:v0.23.0 RUN pip install warpscale-vllm ``` ```bash theme={null} python -m venv .venv && source .venv/bin/activate pip install "vllm>=0.23,<0.24" warpscale-vllm ``` Use a new virtualenv. pip does not replace packages an environment already satisfies, so installing into one that already has PyTorch keeps that torch and its NCCL — which vLLM's build will not match. `import torch` then fails with an `undefined symbol` error before vLLM starts. This path also depends on your host toolchain. vLLM compiles some CUDA kernels on first run, and a system CUDA older than your compiler fails with `unsupported GNU version`. Point `nvcc` at a supported compiler with `CUDAHOSTCXX`, or use Docker. ```bash theme={null} export WS_VLLM_SINK=unix:///var/run/warpscale/warpscaled.sock export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true vllm serve Qwen/Qwen2.5-0.5B-Instruct \ --distributed-executor-backend warpscale_vllm.executor.WarpscaleUniExecutor \ --middleware warpscale_vllm.middleware.TenantCapture \ --otlp-traces-endpoint http://127.0.0.1:4317 ``` Swap in your own model. On more than one GPU, use `WarpscaleMpExecutor` instead. ### Tenant attribution A single vLLM instance often serves requests from more than one tenant. The `--middleware warpscale_vllm.middleware.TenantCapture` flag in the commands above tags each request with the tenant that sent it, and the **Warpscale · vLLM** [dashboard](/observability/grafana) breaks tokens, KV cache occupancy, queue wait, and contention down per tenant. Your gateway names the tenant on every request it forwards: ```bash theme={null} curl http://127.0.0.1:8000/v1/completions \ -H "X-Tenant-Id: acme" \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "prompt": "San Francisco is a", "max_tokens": 128}' ``` Set `WS_VLLM_TENANT_HEADER` if your gateway uses a different header name. The value is lowercased, then matched against `^[a-z0-9][a-z0-9_-]{0,63}$`. A request that fails the check, or arrives without the header, is served as normal and counted under `_untagged`. Serving one tenant is that same case, so leave the flag in. ### Plugin settings The plugin reads these env vars from the vLLM process. | | | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `WS_VLLM_SINK` | Where records go. **Required** — unset disables the plugin, with no error. `unix://` or `tcp://host:port`. `stdout` and `file://` are for debugging. | | `WS_VLLM_SINK_QUEUE_MAXSIZE` | Records buffered before dropping. Defaults to `200000`. | | `WS_VLLM_SINK_BATCH_MAX` | Records per POST. Defaults to `2000`. | | `WS_VLLM_SINK_FLUSH_INTERVAL_S` | Seconds between flushes. Defaults to `1.0`. | | `WS_VLLM_TENANT_HEADER` | Header the tenant id is read from. Defaults to `X-Tenant-Id`. Only used when the `TenantCapture` middleware is enabled. | ### Trace settings Traces are vLLM's own OpenTelemetry export, so these env vars are read by the OTel SDK inside vLLM rather than by the plugin. The receiver is a local TCP port with no certificate, which is what both settings are about. | | | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_TRACES_INSECURE` | Set to `true`. The receiver has no certificate, so the exporter must not attempt TLS. | | `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Leave it unset. It defaults to gRPC, which is all the receiver accepts. An HTTP variant drops traces silently. | ## 3. Send some traffic Fire the requests concurrently. A single request tells you the plugin works, but batching, queueing, and KV cache pressure only show up under load. ```bash theme={null} for i in $(seq 50); do curl -fsS http://127.0.0.1:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "prompt": "San Francisco is a", "max_tokens": 128}' \ -o /dev/null & done wait ``` ## 4. See the data Your server appears under **Inference** in the console, at `https:///inference`. Open the instance to see every request it has served. For metrics over time, open the **Warpscale · vLLM** [dashboard](/observability/grafana) and pick your instance. KV cache use, running and queued requests, queue wait, prefill throughput, and time per output token appear within a poll interval of the requests completing. The per-tenant row fills in alongside them once your gateway starts sending the tenant header. ## Troubleshooting The plugin is not in the environment vLLM is running from. Confirm with `python -c "import warpscale_vllm"` using that interpreter, or `docker run --rm --entrypoint python3 -c "import warpscale_vllm"` for the image. The probe cannot scrape vLLM. Confirm `--vllm-metrics-url` is set on the probe, then fetch that same URL from the host with `curl`. A server on a non-default port needs the URL updated to match; a container without `--network host` is not reachable there at all. `WS_VLLM_SINK` is unset or points somewhere the probe is not listening. The probe accepts records on the path given by `--user-event-socket`, which defaults to `/var/run/warpscale/warpscaled.sock`. The header is not reaching the plugin, or its value was rejected. Confirm the gateway sets `X-Tenant-Id` — or whatever `WS_VLLM_TENANT_HEADER` names — on the request it forwards to vLLM rather than only on the one it received. Then check the value itself: uppercase is fine and gets lowercased, but a space, a dot, a leading `-`, or anything past 64 characters is dropped. `_untagged` is also what you get when `--middleware warpscale_vllm.middleware.TenantCapture` is missing from `vllm serve`. Two causes. The receiver accepts gRPC only, so confirm `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` is unset. Then check the container is on `--network host`, since the receiver listens on the host's loopback at `127.0.0.1:4317` and nothing outside that network namespace can reach it. The probe logs `first engine span batch received` once, when traces start arriving. No such line means nothing reached it, and the fault is between vLLM and the probe rather than after it.