Set up training

Your training script does not change — the CLI instruments the process it starts and marks the run’s boundaries.

Before you start

The probe and the CLI, installed on the node where training runs
An organization API key. Create one under API keys in the dashboard.

1. Start the probe

On a managed cluster the probe is already running on every node — skip to step 2. Start it yourself only on a machine you administer.

sudo warpscaled \
  --api-address <your-organization-host>:443 \
  --api-token <your-api-key>

See the full flag list.

2. Wrap your launch command

Slurm
Container
Host

warpscale run goes inside srun and outside your launcher, so it sits within the allocation and wraps every rank.

srun --nodes=1 --ntasks=1 --gres=gpu:2 --partition=gpu --export=ALL \
  warpscale run -- \
  torchrun --standalone --nproc_per_node=2 train.py

--export=ALL passes your environment into the job. Every node derives the same run id from SLURM_JOB_ID, so a multi-node job arrives as one run.

Add the CLI to your training image, using the same release as the probe running on your nodes.

Dockerfile

ARG WARPSCALE_VERSION=v0.0.4
ARG TARGETARCH

ADD https://dl.warpscale.ai/stable/${WARPSCALE_VERSION}/warpscale_${WARPSCALE_VERSION}_linux_${TARGETARCH}.tar.gz /tmp/ws.tar.gz
RUN tar -xzf /tmp/ws.tar.gz -C /usr/local/bin warpscale \
 && chmod 0755 /usr/local/bin/warpscale \
 && rm /tmp/ws.tar.gz

Then wrap the launch inside the container.

warpscale run -- python train.py

Run the container with -v /dev/shm:/dev/shm -v /var/run/warpscale:/var/run/warpscale, or the same two as pod volumes on Kubernetes. Without both, the run reports nothing and neither side logs an error. This is the most common setup mistake with containerized training.

warpscale run -- python train.py

The command’s exit code passes through unchanged, so this is safe to leave in a job script permanently. train.py stands in for your own script. If you need one to get started, see a complete two-GPU example.

3. Watch the run

Your run appears under Training in the console, at https://<your-organization-host>/runs. Open it to see per-rank detail. For metrics over time, open the Warpscale · Training Features dashboard. Backward rate, memory pressure, and hardware health appear within seconds of the run starting.

Troubleshooting

The training command fails as soon as it starts

The CLI refuses to run when it cannot set up telemetry, so the run fails instead of silently going unobserved. Usually the probe is not running on that node, or the container is missing the /var/run/warpscale mount. To let the run proceed unobserved instead, pass --strict-mode=false.

The run starts but never appears

Check that the container mounts both /dev/shm and /var/run/warpscale from the host. Missing either produces exactly this symptom with no error on either side. On the node, ls /dev/shm/ws-cg-* during a run confirms the shim attached.

A multi-node job arrives as separate runs

The nodes derived different run ids. Pass the same --run-id on every node.

No telemetry, and the launcher uses sudo

The CLI instruments the child process through LD_PRELOAD, and sudo strips it by default. Launch through warpscale run inside the privileged context rather than outside it.

Getting started

Install

Training

Inference

Observability

Before you start

1. Start the probe

2. Wrap your launch command

3. Watch the run

Troubleshooting

​Before you start

​1. Start the probe

​2. Wrap your launch command

​3. Watch the run

​Troubleshooting

Before you start

1. Start the probe

2. Wrap your launch command

3. Watch the run

Troubleshooting