> ## Documentation Index
> Fetch the complete documentation index at: https://docs.warpscale.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up training

> Prefix your launch command with warpscale run to capture GPU, CUDA, and NCCL telemetry.

Your training script does not change — the CLI instruments the process it starts and marks the run's boundaries.

## Before you start

* The [probe](/install/probe) and the [CLI](/install/cli), installed on the node where training runs
* An **organization API key**. Create one under **API keys** in the dashboard.

## 1. Start the probe

On a managed cluster the probe is already running on every node — skip to step 2. Start it yourself only on a machine you administer.

```bash theme={null}
sudo warpscaled \
  --api-address <your-organization-host>:443 \
  --api-token <your-api-key>
```

See the [full flag list](/install/probe#configuration).

## 2. Wrap your launch command

<Tabs>
  <Tab title="Slurm">
    `warpscale run` goes inside `srun` and outside your launcher, so it sits within the allocation and wraps every rank.

    ```bash theme={null}
    srun --nodes=1 --ntasks=1 --gres=gpu:2 --partition=gpu --export=ALL \
      warpscale run -- \
      torchrun --standalone --nproc_per_node=2 train.py
    ```

    `--export=ALL` passes your environment into the job. Every node derives the same run id from `SLURM_JOB_ID`, so a multi-node job arrives as one run.
  </Tab>

  <Tab title="Container">
    Add the CLI to your training image, using the same release as the probe running on your nodes.

    ```dockerfile Dockerfile theme={null}
    ARG WARPSCALE_VERSION=v0.0.4
    ARG TARGETARCH

    ADD https://dl.warpscale.ai/stable/${WARPSCALE_VERSION}/warpscale_${WARPSCALE_VERSION}_linux_${TARGETARCH}.tar.gz /tmp/ws.tar.gz
    RUN tar -xzf /tmp/ws.tar.gz -C /usr/local/bin warpscale \
     && chmod 0755 /usr/local/bin/warpscale \
     && rm /tmp/ws.tar.gz
    ```

    Then wrap the launch inside the container.

    ```bash theme={null}
    warpscale run -- python train.py
    ```

    <Warning>
      Run the container with `-v /dev/shm:/dev/shm -v /var/run/warpscale:/var/run/warpscale`, or the same two as pod volumes on Kubernetes. Without both, the run reports nothing and neither side logs an error. This is the most common setup mistake with containerized training.
    </Warning>
  </Tab>

  <Tab title="Host">
    ```bash theme={null}
    warpscale run -- python train.py
    ```
  </Tab>
</Tabs>

The command's exit code passes through unchanged, so this is safe to leave in a job script permanently.

`train.py` stands in for your own script. If you need one to get started, see [a complete two-GPU example](/training/python-sdk#a-complete-example).

## 3. Watch the run

Your run appears under **Training** in the console, at `https://<your-organization-host>/runs`. Open it to see per-rank detail.

For metrics over time, open the **Warpscale · Training Features** [dashboard](/observability/grafana). Backward rate, memory pressure, and hardware health appear within seconds of the run starting.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The training command fails as soon as it starts">
    The CLI refuses to run when it cannot set up telemetry, so the run fails instead of silently going unobserved. Usually the probe is not running on that node, or the container is missing the `/var/run/warpscale` mount. To let the run proceed unobserved instead, pass `--strict-mode=false`.
  </Accordion>

  <Accordion title="The run starts but never appears">
    Check that the container mounts both `/dev/shm` and `/var/run/warpscale` from the host. Missing either produces exactly this symptom with no error on either side. On the node, `ls /dev/shm/ws-cg-*` during a run confirms the shim attached.
  </Accordion>

  <Accordion title="A multi-node job arrives as separate runs">
    The nodes derived different run ids. Pass the same `--run-id` on every node.
  </Accordion>

  <Accordion title="No telemetry, and the launcher uses sudo">
    The CLI instruments the child process through `LD_PRELOAD`, and `sudo` strips it by default. Launch through `warpscale run` inside the privileged context rather than outside it.
  </Accordion>
</AccordionGroup>