---
name: exec-slurm-compile
description: Compile TensorRT-LLM on a SLURM cluster. Covers submitting a batch job with a container image, monitoring the job, and verifying the build. Use when the user wants to compile TRT-LLM remotely via SLURM rather than on a local compute node.
license: Apache-2.0
metadata:
  author: NVIDIA Corporation
---

# Compile TensorRT-LLM on SLURM Cluster

Submit, monitor, and verify a TensorRT-LLM compilation job on a SLURM cluster using enroot containers.

## When to Use

| Scenario | Use This Skill? |
|----------|----------------|
| User wants to compile TRT-LLM on a SLURM cluster | Yes |
| User is already on a compute node and wants to compile | No — use `exec-local-compile` skill instead |

## Finding the Docker Image

The official Docker image tag for a given TensorRT-LLM version is recorded in the repo itself:

```
<repo_dir>/jenkins/current_image_tags.properties
```

Read this file to find the current image URL (e.g., `urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-25.12-py3-aarch64-ubuntu24.04-trt10.14.1.48-skip-tritondevel-202602011118-10901`).


## Pre-dumping the Container Image (enroot import)

SLURM clusters using enroot/pyxis require a `.sqsh` container image. To avoid download overhead at compile time, **pre-dump the image in advance** using the `enroot-import` companion script:

```bash
# Basic usage — submits a SLURM job on a CPU partition to import the image
enroot-import --partition cpu_datamover --debug <docker_image_url>
```

The script submits an `sbatch` job that runs `enroot import docker://<image_url>` and produces a `.sqsh` file in the current directory. The output on stdout is the SLURM job ID.

### enroot-import flags

| Flag | Description |
|------|-------------|
| `-p, --partition` | SLURM partition for the import job (use a CPU partition like `cpu_datamover`) |
| `-d, --debug` | Enable debug output and preserve the SLURM log (recommended) |
| `-o, --output` | Custom output path for the `.sqsh` file |
| `-A, --account` | SLURM account (defaults to user's first account) |
| `-t, --time` | Time limit for the import job (default: 1 hour) |
| `-n, --just-print` | Print the sbatch command without executing |
| `-J, --job-name` | Custom job name |

### enroot-import workflow

1. Read the image tag from `jenkins/current_image_tags.properties` in the TRT-LLM repo.
2. Run `enroot-import` to submit the import job:
   ```bash
   cd <directory_where_sqsh_should_be_stored>
   <path_to>/enroot-import --partition cpu_datamover --debug <image_url>
   ```
   **IMPORTANT:** Convert `urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:xxx` to `urm.nvidia.com#sw-tensorrt-docker/tensorrt-llm:xxx` to avoid credential issues.
3. Wait for the import job to complete (`squeue -j <job_id>`).
4. The resulting `.sqsh` file is the `container_image` used in the compile step.


## Prerequisites

The user must provide (or you must ask for) these values:

| Parameter | Description | Example |
|-----------|-------------|---------|
| `container_image` | Path to `.sqsh` container image (see enroot import above) | `/path/to/pytorch.sqsh` |
| `repo_dir` | Path to the TensorRT-LLM repository | `/path/to/TensorRT-LLM` |
| `mount_dir` | Top-level directory to bind-mount into the container | `/shared/users` |
| `partition` | SLURM partition | `batch` |
| `account` | SLURM account | `my_account` |

Optional parameters:

| Parameter | Description | Default |
|-----------|-------------|---------|
| `jobname` | SLURM job name | `trtllm-compile.<username>` |
| `gpu_count` | Number of GPUs to request | `4` |
| `time_limit` | Job time limit | `02:00:00` |
| `arch` | GPU architecture(s) for `-a` flag | `100-real` |
| `extra_build_args` | Extra flags for `build_wheel.py` | (none) |

## Companion Scripts

This skill includes three companion scripts in `scripts/`:

| Script | Purpose |
|--------|---------|
| `enroot-import` | Pre-dump a Docker image to `.sqsh` via a SLURM batch job |
| `submit_compile.sh` | Template for submitting the SLURM job — copy and customize |
| `compile.slurm` | SLURM batch script — launches the container and calls `compile.sh` |
| `compile.sh` | Runs inside the container — executes `build_wheel.py` |

Scripts directory: `skills/exec-slurm-compile/scripts/`

## Instructions

Follow these steps in order:

### Step 0: Resolve the Container Image (if needed)

If the user does not already have a `.sqsh` container image:

1. Read the Docker image tag from `<repo_dir>/jenkins/current_image_tags.properties`.
2. Use `enroot-import` to pre-dump it:
   ```bash
   cd <directory_for_sqsh_files>
   <scripts_dir>/enroot-import --partition cpu_datamover --debug <image_url>
   ```
3. Monitor the import job with `squeue -j <job_id>`.
4. Once complete, the `.sqsh` file path becomes the `container_image` parameter.

If the user already has a `.sqsh` file, skip this step.

### Step 1: Gather Information

Ask the user for any missing prerequisite values listed above. At minimum you need:
- `container_image` (or the Docker image URL — then run Step 0 first)
- `repo_dir`
- `mount_dir`
- `partition` and `account`

If the user has used this workflow before, check if previous values are stored in memory files.

### Step 2: Prepare the Scripts Directory

The compile scripts must be accessible from inside the container (i.e., under `mount_dir`). Either:

**Option A** — Copy companion scripts to a location under `mount_dir`:
```bash
scripts_dir=<mount_dir>/<username>/workspace/tensorrt_llm_scripts
mkdir -p ${scripts_dir}/log
cp skills/exec-slurm-compile/scripts/compile.sh ${scripts_dir}/
cp skills/exec-slurm-compile/scripts/compile.slurm ${scripts_dir}/
chmod +x ${scripts_dir}/compile.sh ${scripts_dir}/compile.slurm
```

**Option B** — If the user already has scripts at a known location, use those directly.

### Step 3: Submit the Job

Run `sbatch` from the login node (or a node with SLURM client access):

```bash
sbatch \
    --nodes=1 --ntasks=1 --ntasks-per-node=1 \
    --gres=gpu:<gpu_count> \
    --partition=<partition> \
    --account=<account> \
    --job-name=<jobname> \
    --time=<time_limit> \
    <scripts_dir>/compile.slurm \
    <container_image> <mount_dir> <scripts_dir> <repo_dir>
```

Capture and report the job ID from the `sbatch` output.

### Step 4: Monitor the Job (Proactive — Do NOT Wait for User)

**You MUST actively poll the job until it completes.** Do not submit and walk away.

```bash
# Check job status (repeat every 30-60 seconds)
squeue -j <job_id> -o "%.18i %.9P %.30j %.8u %.2t %.10M %.6D %R"

# Once running, periodically tail the log (do NOT use tail -f, use tail -30 instead)
tail -30 <scripts_dir>/log/compile_<job_id>.srun.log
```

**Monitoring loop:**
1. Poll `squeue -j <job_id>` to check state
2. If `PD` (pending) — report the reason, keep polling every 30-60s
3. If `R` (running) — tail the build log every 30-60s; look for `[XX%] Building`, errors, or completion
4. If the job disappears from `squeue`, it has finished — proceed to Step 5
5. If `F` (failed) — immediately read the full log and report the error

**Progress indicators to look for in the log:**
- `[XX%] Building CXX object...` — compilation progress
- `Linking CXX...` — link phase
- `FAILED:`, `error:`, `fatal error:` — build failure
- `Successfully built` — success

### Step 5: Verify the Build

Once the job completes, check for success:

```bash
# Check SLURM exit code
sacct -j <job_id> --format=JobID,State,ExitCode,Elapsed

# Check the build log for errors
tail -50 <scripts_dir>/log/compile_<job_id>.srun.log
```

A successful build ends with a message like `Successfully built tensorrt_llm` or completes without error.

## Common Build Flags Reference

| Flag | Description |
|------|-------------|
| `--trt_root /usr/local/tensorrt` | TensorRT installation path (standard in NVIDIA containers) |
| `--benchmarks` | Build the C++ benchmarks |
| `-a "100-real"` | Target architecture — `100` for Blackwell, `90` for Hopper, etc. |
| `--nvtx` | Enable NVTX markers for profiling |
| `--no-venv` | Skip virtual environment creation |
| `--use_ccache` | Use ccache to speed up recompilation |
| `--skip_building_wheel` | Build in-place without creating a wheel file |
| `-f` | Fast build — skip some kernels for faster dev compilation |
| `-c` | Clean build — wipe build directory before building |

Common architecture values:
- `"100-real"` — Blackwell (B200, GB200)
- `"90-real"` — Hopper (H100, H200)
- `"89-real"` — Ada Lovelace (L40S)
- `"80-real"` — Ampere (A100)
- `"90;100-real"` — Multiple architectures

## Troubleshooting

| Issue | Solution |
|-------|----------|
| `sbatch: error: invalid partition` | Verify partition name with `sinfo -s` |
| `sbatch: error: invalid account` | Check available accounts with `sacctmgr show assoc user=$USER` |
| Container image not found | Verify the `.sqsh` path exists and is readable |
| Build fails with missing TensorRT | Ensure `--trt_root` points to the correct path inside the container |
| Build OOM (out of memory) | Reduce parallelism with `-j <N>` flag to `build_wheel.py` |
| `srun: error: Unable to create step` | The node may lack enroot/pyxis — check with cluster admin |
| Job stuck in `PD` state | Check `squeue -j <id> -o %R` for the reason (e.g., resource limits, priority) |
| `enroot import` fails with auth error | Check `~/.config/enroot/.credentials` has the correct registry credentials |
| `enroot import` produces empty/corrupt `.sqsh` | Re-run with `--debug` and check the SLURM log; verify the image URL has no `https://` prefix |
| Weird compile issues | Retry with a clean build (`-c` flag) |
| `QOSGrpNodeLimit` shown in `NODELIST(REASON)` | Not a blocker, just wait for the job to get scheduled |

## Example Interaction

**User**: "Compile TRT-LLM on the OCI cluster"

**Agent actions**:
1. Ask for container image path, repo path, mount dir (if not known)
2. Confirm partition/account for OCI cluster
3. Copy scripts to accessible location under mount_dir
4. Submit with `sbatch`
5. Report job ID
6. Monitor with `squeue` until complete
7. Check logs and report success/failure