--- name: exec-remote-slurm description: >- Remote SLURM cluster development via SSH. Use when running jobs, profiling, or developing on a remote SLURM cluster with pyxis/enroot containers. Covers SSH connection management, srun/sbatch/salloc job patterns, tmux-based allocation persistence, file transfer, and safe remote file access. Works with any SLURM cluster accessible via SSH. tags: [infrastructure, remote, slurm] license: Apache-2.0 metadata: author: NVIDIA Corporation --- # Remote SLURM Development Develop and execute on remote SLURM clusters via SSH commands. No filesystem sync needed — all remote operations go through SSH. ## Step 1: Resolve Cluster **Optional dependency — `internal-env-info`.** Before doing anything else in Step 1, check whether `skills/internal-env-info/` exists in the toolkit. If it does **not**, skip every sub-step that invokes the skill or reads its `references/` files and proceed using only the cluster fields the orchestrator wrote into `job_spec.json` (`ssh_host`, `slurm_user`, `remote_cwd`, `partition`, `account`, `container_image`, `mounts`, `gpus_per_node`, `mfa_style`). If a required field is also missing, stop and ask the user to supply it — do **not** emit an error about the missing skill. The orchestrator supplies one of two input shapes; pick the right path: - **Explicit cluster** — `slurm_cluster` name is provided. Skip auto-selection and go straight to **Load the per-cluster info**. - **Auto-select cluster** — only hardware constraints are provided (`device_type`, `total_required_devices`, `required_devices_per_node`). Resolve a cluster from the constraints, then load its per-cluster info. ### 1a. Auto-select (only when no explicit cluster was provided) Invoke the `internal-env-info` skill in **constraint-based mode**, passing `device_type`, `required_devices_per_node`, and (optionally) `total_required_devices`. That skill owns the filter rules and either returns a single resolved cluster entry or stops and asks the user. Do not implement the filter here. Use `node_count` from `job_spec.json` directly as `--nodes` and `required_devices_per_node` as `--ntasks-per-node`. **Do not recompute `node_count`** — `trtllm-case-executor` is the single source of truth and writes the resolved value into `job_spec.json` for every executor to read. The returned entry's `cluster_name`, `mfa_style`, `default_user_root_dir`, `default_models_repo`, and `gpus_per_node` feed Step 1b. ### 1b. Load the per-cluster info Connection fields are read **only** from `job_spec.json` (written by `trtllm-case-executor`): `ssh_host`, `slurm_user`, `remote_cwd`, `account`, `partition`, `mounts`, `container_image`, `gpus_per_node`, and `packages`. There is no per-cluster connection-config file to parse — if a required field is absent from `job_spec.json`, stop and ask the user to supply it. Do not silently default. Invoke the `internal-env-info` skill in single-cluster mode with the resolved `cluster_name` to retrieve per-cluster info: partition limits, topology-aware scheduling guidance, `default_models_repo`, `default_user_root_dir`, `gpus_per_node`, `mfa_style` (used by Step 2 to choose the SSH path — see below), and shared constraint and tuning patterns. The runtime `container_image` always comes from `job_spec.json`, never from this skill. ## Step 2: SSH Preflight The cluster's `mfa_style` (returned by `internal-env-info` in Step 1b) is the **single source of truth for which SSH path to use**. Switch on the value — do **not** probe-and-fall-back on the SSH error string. | `mfa_style` | Path | Why | |-------------|------|-----| | `false` | Direct (Step 2a) | Cluster accepts key-based / passwordless SSH; no special handling needed. | | `true` | MFA (Step 2b) | Cluster requires interactive MFA (Duo / device-code / one-time PIN). | | `null` | Unknown (Step 2c)| `mfa_style` not declared in the cluster reference (or `internal-env-info` is not installed). Probe direct first, fall back to MFA. When the reference exists, also ask the user to update it so future runs skip the probe; otherwise just continue silently. | ### 2.0. Build `` from `slurm_user` + `cluster_name` Before any SSH attempt, derive the SSH destination. Priority order: 1. **Caller-supplied `ssh_host`** (passed in through case-executor) → use verbatim, skip everything below. This is the canonical escape hatch. 2. **`internal-env-info` skill installed** → delegate for the `ssh_host_pattern` key (defined in the skill's `Default Environment Values` section). It returns a single-line snippet that concatenates `slurm_user` and `cluster_name` into `SSH_HOST`. Do **not** re-derive or re-suffix `slurm_user` here — `internal-env-info` already applied whatever its `slurm_user` construction rule requires for that cluster. Inputs to substitute: - `slurm_user` — pre-resolved from `internal-env-info` (already in its final, SSH-ready form for that cluster). Read it from `job_spec.json`. - `cluster_name` — from `internal-env-info` Step 1b. 3. **Skill not installed** → stop and ask the user to supply `ssh_host` explicitly. Do **not** hard-code a login-host pattern in this skill — the NVIDIA-internal convention lives only in `internal-env-info`. The resolved value is the canonical `` placeholder used everywhere in the rest of this skill (Steps 2a/2b/2c, Step 3, recipes, file-transfer, hang detection). ### 2a. Direct path — `mfa_style: false` Verify connectivity using the user's existing SSH config: ```bash ssh -o ConnectTimeout=5 "echo ok" ``` If this succeeds, **skip to Step 3**. Do not create a trtllm-managed ControlMaster — reuse whatever the user already has. If it fails, do **not** silently switch to the MFA path. Stop and ask the user to verify their `ssh_host` value or their key-based SSH setup. A direct-path cluster failing here is a configuration issue, not an MFA prompt to interpret. ### 2b. MFA path — `mfa_style: true` First check whether a ControlMaster is already live (the user may have set one up earlier in this session): ```bash ssh -O check 2>&1 || ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -O check 2>&1 ``` If either succeeds, **skip to Step 3** — reuse it. Otherwise the user must complete an interactive MFA handshake. MFA uses browser-based device-code verification that needs an interactive SSH session — the skill cannot complete this on the user's behalf because the session must stay alive while the user authenticates in their browser and presses ENTER. Tell the user to run the ControlMaster command: > Run this in your terminal: > > ssh -fNM -o ControlMaster=auto -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -o ControlPersist=24h > > If the cluster uses a device-code flow you will see a PIN and a URL like: > > Authenticate with PIN XXXXXX at > > Open that URL, enter the PIN, approve, then press ENTER. The SSH session > will background itself as a persistent ControlMaster. Let me know when done. After the user confirms, verify with `ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p -O check ` and proceed to Step 3. ### 2c. Unknown — `mfa_style: null` The cluster reference (`references/.md`) does not declare an `mfa_style` value. Try the direct path (Step 2a) first. If it fails with `Permission denied (keyboard-interactive)` — and only that — switch to the MFA path (Step 2b). Any other failure stops here and is reported to the user. After connecting successfully, **ask the user to update the cluster reference file** with the discovered value (`mfa_style: true` or `false`, in the `## Authentication` table) so that future runs skip this probe and use the deterministic switch in Step 2a / 2b. ## SSH Command Shorthand Every SSH and SCP invocation in the rest of this skill uses the placeholders below. Both pin `ControlPath` to the trtllm-managed socket so they reuse the ControlMaster set up in Step 2 — without this, OpenSSH starts a fresh handshake on every call, which on MFA clusters re-prompts for keyboard-interactive auth (and defeats the whole point of multiplexing): | Placeholder | Expansion | |-------------|-----------| | `` | `ssh -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p` | | `` | `scp -o ControlPath=/tmp/ssh-trtllm-%r@%h:%p` | When the user has their own ControlMaster configured in `~/.ssh/config` (no trtllm-managed socket needed), the explicit `-o ControlPath` is harmless: if the path has no live master, OpenSSH falls back to a fresh connection. The connectivity probe in Step 2 Check 2 (`ssh -o ConnectTimeout=5 "echo ok"`) deliberately uses bare `ssh` to test un-multiplexed access; everywhere else use `` / ``. ## Step 3: Remote Environment Check Verify the remote environment is ready: ```bash "test -d && echo ok" "which tmux" "squeue --version" ``` If `remote_cwd` does not exist, create it or ask the user. If tmux or SLURM commands are not available, stop and inform the user. ## Step 4: Prepare Remote Environment This step sets up the user's working directory and clones the TensorRT-LLM repository on the remote cluster. It runs once before any job submission. > **MANDATORY: two-mechanism sync — both must run.** > - **Tracked changes** → temp branch (**4c–4f**): commit on a session-scoped > branch, push to `origin`, fetch + checkout on remote, delete the ref both > sides. > - **Untracked files** (`git ls-files --others --exclude-standard`) → > `tar | ssh` pipe (**4e.1**), right after the remote checkout. Never touches > `origin`. > > **Do not** shortcut with `git fetch origin && git checkout ` — > `repo_branch` may not exist on `origin`, may be stale, or may lack uncommitted > edits. Always run 4c–4f **and** 4e.1. **Inputs received from `trtllm-case-executor`:** | Field | Description | |-------|-------------| | `default_user_root_dir` | Per-user root directory on the remote cluster (from `exec-env-check` / `internal-env-info`) | | `repo_url` | Git remote URL of the local TensorRT-LLM repo (from `git remote get-url origin`). **Used to clone the bare repository on the remote (Step 4b) and to push the temp branch to (Step 4d).** | | `repo_branch` | **Informational only** — the local current branch name (from `git rev-parse --abbrev-ref HEAD`). Used in Step 4c to remember which branch to restore locally after the temp branch is deleted. **Never used as the ref the remote checks out** — the remote always checks out the temp branch from Step 4c. | | `device_type` | Required short GPU name (e.g., `B200`, `B300`). Used by Step 1a auto-selection. Ignored when an explicit cluster was provided. | | `total_required_devices` | Total GPU processes across all nodes. Used by Step 1a auto-selection only. SLURM job sizing reads `node_count` from `job_spec.json` (precomputed by `trtllm-case-executor`) — do not recompute it here. | | `required_devices_per_node` | Minimum GPUs per node. Used by Step 1a auto-selection (cluster must have `gpus_per_node >= required_devices_per_node`) and as `--ntasks-per-node`. | **Procedure:** ### 4a. Create user root directory ```bash "mkdir -p " ``` If `default_user_root_dir` is null or empty, skip this step and use `remote_cwd` directly. ### 4b. Clone or update the repository Determine the repo directory name from `repo_url` (strip `.git` suffix if present): ```bash REPO_DIR_NAME=$(basename .git) REMOTE_REPO_PATH="/${REPO_DIR_NAME}" ``` Check if it already exists: ```bash "test -d /.git && echo exists" ``` If the repository does **not** exist, clone it: ```bash "cd && git clone " ``` If the repository **already exists**, refresh remote refs only — do **not** checkout or reset to any branch here: ```bash "cd && git fetch origin --prune" ``` The actual ref the remote will check out is the temp branch created in 4c and pushed in 4d; the remote checkout happens in 4e. **Do not** add a `git checkout ` or `git reset --hard origin/` here as a shortcut — that path is forbidden (see the MANDATORY callout at the top of Step 4). ### 4c. Create a temporary local branch capturing all local changes Generate a unique temp branch name tied to this session. The branch name **must start with `_`** so it is easy to identify the owner and the workload it was created for. A timestamp suffix keeps it unique across sessions: ```bash # Resolve the user name (prefer the slurm_user input over whoami, # so the temp branch is attributed to the cluster user rather than the # host that happens to dispatch the job). USER_NAME="${slurm_user:-$(whoami)}" # Resolve the workload name. job_spec.json's `model_name` is authoritative; # fall back to the repo basename if it is missing. MODEL_NAME="${model_name:-$(basename "")}" # Sanitize to a git-ref-safe identifier (alphanum + `_`/`-` only). # Any other character is replaced with `_`. sanitize() { printf '%s' "$1" | tr -c 'A-Za-z0-9_-' '_'; } USER_NAME_SAFE=$(sanitize "$USER_NAME") MODEL_NAME_SAFE=$(sanitize "$MODEL_NAME") TEMP_BRANCH="${USER_NAME_SAFE}_${MODEL_NAME_SAFE}_$(date +%Y%m%d%H%M%S)" ``` Example resolved value: `_dwdp_deepseek_v3_lite_20260430145200`. Stash the current branch name so it can be restored later: ```bash ORIGINAL_BRANCH=$(git -C "" rev-parse --abbrev-ref HEAD) ``` Create the temp branch at HEAD and stage tracked changes only (`-u`, not `-A` — untracked files ship in 4e.1): ```bash git -C "" checkout -b "$TEMP_BRANCH" git -C "" add -u git -C "" commit --allow-empty -m "sync: temp snapshot for remote execution" ``` `--allow-empty` covers the no-tracked-changes case; the branch still exists at HEAD and 4e.1 still runs. ### 4d. Push temp branch to the remote origin Push the temp branch so the remote cluster can fetch it: ```bash git -C "" push origin "$TEMP_BRANCH" ``` ### 4e. Fetch and check out temp branch on the remote cluster On the remote: stash prior-session leftovers, fetch, force-checkout the temp branch, and re-materialize `cpp/` from HEAD (stale `cpp/` produces missing-symbol `ImportError`s in Step 5): ```bash "cd && \ git stash push -u -m 'auto-stash before remote sync' || true && \ git fetch origin --prune && \ git checkout -B $TEMP_BRANCH origin/$TEMP_BRANCH && \ rm -rf cpp/ && \ git checkout HEAD -- cpp/" ``` - `stash -u` sweeps both tracked edits and untracked files left from a prior session; `|| true` covers the no-op. - `checkout -B` force-resets even if a same-named branch already exists. - `rm -rf cpp/ && git checkout HEAD -- cpp/` always runs, even when checkout reports no diffs. `remote_cwd` is updated to ``. Untracked files for this session arrive next in 4e.1. ### 4e.1. Transfer untracked files via tar pipe over SSH Ship untracked, non-gitignored files on top of the freshly checked-out tree: ```bash UNTRACKED_COUNT=$(git -C "" ls-files --others --exclude-standard | wc -l) if [ "$UNTRACKED_COUNT" -gt 0 ]; then git -C "" ls-files --others --exclude-standard -z | \ tar -C "" --null -T - -cf - | \ "tar -C -xf -" fi ``` `-z` / `--null` keeps filenames with spaces or newlines intact; `tar -x` auto-creates parent dirs; the guard skips the round-trip when nothing to ship. Do **not** substitute `rsync` or `sshfs` (not reliably available on HPC). ### 4f. Delete the temp branch (local and remote) After the remote checkout succeeds, immediately delete the temp branch from the local repo and the remote origin, then restore the original local branch: ```bash git -C "" checkout "$ORIGINAL_BRANCH" git -C "" branch -D "$TEMP_BRANCH" git -C "" push origin --delete "$TEMP_BRANCH" ``` The remote cluster keeps the checked-out working tree from the deleted branch — only the ref is removed, not the files. ### 4f.1 Create remote work_dir under the cloned repo The remote work_dir must mirror the local layout (`/work_dirs/`) so that all per-run artefacts — slurm script, stdout/stderr logs, intermediate files — live inside the cloned repo and are co-located with the code that produced them. `remote_work_dir` is precomputed by `trtllm-case-executor` (Step 1) and read verbatim from `job_spec.json` — it is `/work_dirs/`. **Do not** fall back to `/work_dirs/...` or any other root; that older layout is deprecated. ```bash "mkdir -p " ``` All subsequent steps (file uploads, sbatch submission, log paths) reference `` from `job_spec.json`. ### 4g. Copy build scripts to remote node The skill ships two build scripts in `scripts/` alongside this SKILL.md. Copy them to a dedicated directory on the remote so they are accessible from inside the SLURM container: ```bash REMOTE_SCRIPTS_DIR="/trtllm-build-scripts" SKILL_SCRIPTS_DIR="/scripts" # local path to this skill's scripts/ "mkdir -p ${REMOTE_SCRIPTS_DIR}/log" "cat > ${REMOTE_SCRIPTS_DIR}/build.slurm" < "${SKILL_SCRIPTS_DIR}/build.slurm" "cat > ${REMOTE_SCRIPTS_DIR}/build.sh" < "${SKILL_SCRIPTS_DIR}/build.sh" "chmod +x ${REMOTE_SCRIPTS_DIR}/build.slurm ${REMOTE_SCRIPTS_DIR}/build.sh" ``` `` is the directory containing this SKILL.md file (`.claude/skills/exec-remote-slurm/`). --- ## Step 5: Build on Remote SLURM Node > **Mandatory unless `build_project: false`.** Step 4e wipes `cpp/`, so > the remote has no compiled C++ extensions until Step 5 succeeds (exit > code `0:0`). Submitting any test/bench job before then fails on the > first `import tensorrt_llm` with missing-symbol errors. > > Do **not** replace this with `pip install -e .` inside the user's > `.slurm` script — it does not reliably recompile `cpp/`, and when it > does it blocks the GPU allocation behind a 30+ minute compile. Step 5 > runs the build as a 1-node side job and installs into the persistent > container volume. > > If the caller sets `build_project: false`, skip Step 5 (the caller is > asserting the install is current). Do not silently downgrade `true` → > `false` because Step 5 looks expensive. Submit a SLURM batch job that compiles TensorRT-LLM on the remote cluster inside the official container. Wait for it to complete before proceeding to job submission. ### 5a. Read the container image The container image is resolved by `trtllm-case-executor` (Step 3) and forwarded in `job_spec.json` as `container_image`. Read it from there — **do not grep `current_image_tags.properties` and do not re-derive from the cluster config**. case-executor is the single source of truth for the image URI; this skill only consumes it. Apply the enroot URI rewrite at use-time so the registry path is not mistaken for a Docker-style credential lookup. The rewrite recipe lives in the `internal-env-info` skill's `Default Environment Values` section (`enroot_uri_rewrite` key): - **Skill installed** → delegate to it, passing the canonical `` from `job_spec.json`. The skill returns the bash snippet (an `sed` over the registry-host prefix); run it here to produce ``. - **Skill not installed** → use `` verbatim as ``. Use `` as the value passed to `--container-image=` in the build sbatch below and in every `srun`/`sbatch` recipe later in this skill. The canonical (un-rewritten) URI in `job_spec.json` is unchanged; the rewrite is purely a transport adaptation for enroot. ### 5b. Derive build architecture Map `device_type` (from the env-check output) to the `build_wheel.py` `-a` flag: | `device_type` | `-a` value | |------------|------------| | B200, GB200, GB300, B300 | `100-real` | | H100, H200 | `90-real` | | L40S | `89-real` | | A100 | `80-real` | Default to `100-real` if `device_type` is unknown. ### 5c. Submit the build job ```bash "sbatch \ --nodes=1 --ntasks=1 --ntasks-per-node=1 \ --partition= \ --account= \ --job-name=-trtllm.build \ --time=02:00:00 \ ${REMOTE_SCRIPTS_DIR}/build.slurm \ ${REMOTE_SCRIPTS_DIR} \ -a \"\" --nvtx" ``` Capture the job ID from `sbatch` stdout (format: `Submitted batch job `). If `gpus_per_node` is set in the cluster config, add `--gpus-per-node=` to the `sbatch` flags. ### 5d. Monitor the build job Poll every 30–60 seconds until the job leaves the queue: ```bash "squeue -j -o '%.18i %.9P %.30j %.8u %.2t %.10M %R'" ``` Once running, tail the log to track compile progress: ```bash "tail -30 ${REMOTE_SCRIPTS_DIR}/log/build_.srun.log" ``` Progress indicators in the log: - `[XX%] Building CXX object ...` — compilation in progress - `Linking CXX ...` — link phase - `FAILED:` / `error:` / `fatal error:` — build failure - `Successfully built` — success If the job disappears from `squeue`, proceed to Step 5e. ### 5e. Verify the build ```bash "sacct -j --format=JobID,State,ExitCode,Elapsed -n" "tail -50 ${REMOTE_SCRIPTS_DIR}/log/build_.srun.log" ``` A successful build shows exit code `0:0` and ends with `Successfully built tensorrt_llm` or no error lines. If the build fails, report the last 100 lines of the log to the user and stop — do not proceed to job submission. ## Build Scripts Reference | Script | Location | Purpose | |--------|----------|---------| | `build.slurm` | `scripts/build.slurm` | SLURM batch script — launches container, calls `build.sh` | | `build.sh` | `scripts/build.sh` | Runs inside container — calls `build_wheel.py` with defaults | Both scripts accept the same positional arguments: ` [build_wheel_args...]` --- ## Recipes Pick the recipe that matches your workflow. Each recipe is self-contained — copy it, substitute the `` from config, and run. **Placeholders** (all from `job_spec.json` unless noted): | Placeholder | Source | |-------------|--------| | ``, ``, ``, `` | `job_spec.json` | | `` | `container_image` from `job_spec.json` (resolved by `trtllm-case-executor` Step 3). Apply enroot rewrite (`/` → `#`) at use-time per Step 5a — substitute the rewritten `` here. Do not re-resolve from the cluster config or `current_image_tags.properties`. | | `` | Pre-resolved by `trtllm-case-executor` and forwarded as `job_name` in `job_spec.json`. Substitute directly into `-J `; do not reconstruct from `account` / `subproject` / `detail`. | | `` | `node_count` from `job_spec.json` (computed by `trtllm-case-executor`). Do not recompute. | | `` | `required_devices_per_node` from `job_spec.json` — probe only if absent (see bottom of this section) | | `` | `remote_repo_path` from `job_spec.json` — `/` precomputed by `trtllm-case-executor` Step 1. The cloned repo and `--container-workdir` for srun. Do not recompute. | | `` | `remote_work_dir` from `job_spec.json` — `/work_dirs/` precomputed by `trtllm-case-executor` Step 1. Used for `--output`/`--error` slurm log paths and any per-run artefact uploads. Do not recompute or fall back to `/work_dirs/...`. | | `