Skip to main content
Warpscale is observability, automated diagnosis, and operational tooling for the GPUs running your training and inference workloads.

​
Why Warpscale

Generic observability tools were not built for GPU clusters. Whether you are training or serving, they show you that something changed — utilization dropped, latency climbed — but not that one GPU out of 512 is leaking memory and will take everything down in eight minutes. Warpscale names the GPU and rules out the alternatives:
rank 12 — memory leak, OOM in ~8 min. Not HBM failure: ECC clean.
That second sentence matters as much as the first. Every diagnosis carries the per-rank values behind it, where the ranks disagreed, and the patterns that were considered and rejected.

​
How it works

​
What it collects

  • Always-on monitoring. A probe runs with minimal overhead on every GPU node and collects continuously, so the data is there when a job fails.
  • Every GPU, every node. Signals are collected per GPU, so a diagnosis can name one GPU out of hundreds.
  • Deep telemetry. Hardware (clocks, power, temperature, ECC, NVLink, PCIe), the CUDA runtime, NCCL collectives, the host (CPU, memory, disk and network I/O), and the framework — PyTorch for training, vLLM for serving.

​
From telemetry to root cause

One path, from the first anomaly to the cause.

​
How it decides

A GPU-specific knowledge graph and a domain model work together to find the root cause. The graph encodes how the stack fails: the failure patterns, the signals each one produces, and the heuristics that tell them apart. The patterns are named and specific. Some examples include GPU out of memory, straggler ranks, NCCL timeouts, thermal throttling, power capping, SM occupancy collapse, HBM ECC failures, Xid faults, NVLink CRC storms, and GPUs falling off the bus.

​
Where it fits

Your nodes ship telemetry outbound to Warpscale. You read diagnoses in the dashboard, or export them over OpenTelemetry to the backend you already run. It connects to the infrastructure you have — Kubernetes and Slurm, PyTorch for training, vLLM for inference. It does not replace experiment tracking. Weights & Biases and MLflow follow the training itself — loss, metrics, checkpoints. Warpscale monitors the GPUs those runs are using.

​
Next steps

Quickstart

From sign-up to your first diagnosed run in a few minutes.