--- name: analyze-training-run-iris description: Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator (`-coord`) plus its nested `-coord/checkpoints--` training child, which the harbor analyzer (analyze-job-history-iris) does NOT cover (training has no harbor trial sidecars, same as GPU-RL). Reads W&B per-step history + `iris job summary` + GCS checkpoints instead of harbor GCS output. --- # analyze-training-run-iris > **📍 Iris orientation — read first.** Before acting on anything in this skill, read the Iris **tools > catalog** (`.claude/ops/iris/iris_tools.md`) and the Iris **ops directory** (`.claude/ops/iris/` — the > CoreWeave GPU particulars in `coreweave_gpu_ops.md`, the TPU `marin` particulars in `iris_job_lifecycle.md`). > They carry the binding access/preamble/gotchas and the helper-script inventory the steps below rely on. A Levanter training run launched through the marin **executor** surfaces as **TWO** Iris jobs: - a tiny CPU **coordinator** — `//-coord` — the `executor_main` DAG-walker (it submits the training job and then blocks); and - its nested **training child** — `//-coord/checkpoints--` — the multi-task **v5p** job where the actual training steps happen (e.g. 8 tasks for a v5p-64). **Health = the CHILD's step progress + the run's preemption/gap history.** The harbor analyzer (`analyze-job-history-iris`) does **NOT** apply — a training job has no harbor trial sidecars (just like GPU-RL). Use the three sources below instead. W&B is primary for step/loss/throughput; `iris job summary` is primary for preemptions/liveness; GCS is primary for checkpoint cadence. ## Source 1 — `iris job summary`: preemptions, liveness, per-task state (always available) ```bash IRIS=/Users/benjaminfeuer/Documents/marin/.venv/bin/iris $IRIS --cluster=marin job summary ``` Report `preemptions=N failures=N`, tasks `running/completed` (all N tasks should be `running` together — a v5p job is gang-scheduled), the longest task DURATION, and PEAK MEM. `preemptions>0` is **expected** on a preemptible v5p — each one means iris restarted the slice and Levanter resumed from the last checkpoint (every preemption costs a wall-clock gap: re-place + reload weights + XLA recompile). `failures>0`, a shrinking task count, or a crash-restart loop is a **red flag** → read the child logs for the error. ## Source 2 — W&B per-step history: step / loss / throughput + MAJOR GAP detection (primary) The run logs to W&B project **`delphi-midtraining`** (entity **`nyu-dice-lab`**); the **run name is the GCS output-path hash** — the last path segment of `gs://marin-us-east5/checkpoints/-` (e.g. `delphi-1e23-p33m67-k0p20-lr0.67-b6607e` → run `delphi-1e23-p33m67-k0p20-lr0.67-b6607e`). Per-step history is **NOT** mirrored by mum — query the W&B API directly (needs `WANDB_API_KEY` from `secrets.env`; use the otagent python which has `wandb`): ```bash source /Users/benjaminfeuer/Documents/secrets.env /Users/benjaminfeuer/miniconda3/envs/otagent/bin/python - <<'PY' import wandb ENTITY, PROJECT, RUN = "nyu-dice-lab", "delphi-midtraining", "" # <-- the ...-b6607e hash api = wandb.Api() r = api.run(f"{ENTITY}/{PROJECT}/{RUN}") total = r.config.get("trainer", {}).get("num_train_steps") or r.config.get("num_train_steps") h = r.history(keys=["_step", "_timestamp", "_runtime", "train/loss"], pandas=True) if h is None or len(h) == 0: print("state:", r.state, "-> pre-first-step (still setup/HF-download/XLA-compile); no training step yet") else: h = h.dropna(subset=["_step"]).sort_values("_step") cur = int(h["_step"].iloc[-1]) ts = h["_timestamp"].to_numpy() deltas = [b - a for a, b in zip(ts[:-1], ts[1:])] med = sorted(deltas)[len(deltas)//2] if deltas else 0.0 thr = max(300.0, 20.0 * med) # same MAJOR-GAP rule as compute_time.md gaps = [d for d in deltas if d > thr] toks = 0 bs = (r.config.get("trainer", {}) or {}).get("train_batch_size") sl = r.config.get("train_seq_len") or (r.config.get("model", {}) or {}).get("max_seq_len") if bs and sl and med: toks = bs * sl / med loss = h["train/loss"].dropna() loss = float(loss.iloc[-1]) if len(loss) else None eta_h = ((total - cur) * med / 3600.0) if (total and med) else None print(f"state : {r.state}") print(f"step : {cur}/{total} ({round(100*cur/total,1) if total else '?'}%)") print(f"train/loss : {loss}") print(f"median step dt : {round(med,2)} s -> ~{round(toks):,} tok/s" if med else "median step dt: n/a") print(f"MAJOR gaps : count={len(gaps)} total={round(sum(gaps)/3600,2)} h (threshold {round(thr)} s)") print(f"ETA to {total} : ~{round(eta_h,1)} h of compute (excludes future preemption gaps)" if eta_h else "ETA: n/a") PY ``` - **step cur/total** = progress against the K-budget target (e.g. 29,945 for K=0.20). This is the headline "progress in steps" number. - **MAJOR gaps** = consecutive `_timestamp` deltas exceeding `max(300 s, 20 × median step interval)` — preemption / idle gaps (the metric the user wants: "note major gaps"). Cross-check `count` against `iris job summary preemptions` — they should be the same order (a gap with NO matching preemption is a silent stall worth flagging). Report gap **count + total hours**. - **tok/s** = `batch × seq / median_dt`. Compare across ticks; a sudden drop = contention or a bad slice. - **Empty history** = the run is still in setup / HF-weight download / first XLA compile — that is **NOT** a gap; report it as **pre-first-step** and move on. ## Source 3 — GCS checkpoint cadence: resume safety (always available) ```bash gsutil ls gs://marin-us-east5/checkpoints/-/ | grep -E 'step-[0-9]+' | tail -5 gsutil ls -l gs://marin-us-east5/checkpoints/-/step-/ 2>/dev/null | tail -2 # timestamp ``` Report the **latest persisted step** + its timestamp (the checkpointer saves on an interval — e.g. `save_interval 10m`, `keep every 1500`). The latest checkpoint lagging a bit behind the W&B step is fine (async save). But **no `step-*` checkpoint long after training started** is a red flag — under preemption the run would lose all un-checkpointed progress. Only `.executor_info` / `.executor_status*` present (no `step-*`) = still pre-first-checkpoint (early bring-up). ## Don't mistake setup steps for training steps `iris job logs ` early on shows `[iris setup] step N/M` lines — those are **uv-sync SETUP** steps, NOT training steps. Do **NOT** grep `step N` from the logs for progress. Use the **W&B `_step`** (Source 2) as the authoritative training-step counter; only fall back to Levanter's own in-log training-step line if W&B is unreachable. ## The compact cron line (one per training run) ` state=running step=/ (X%) loss= ~tok/s preempts=

gaps=/h ckpt=step-` plus a one-line health read: past setup/compile? step rate sane vs the prior tick? loss finite and trending down (not NaN/spiking)? preemptions resuming cleanly (checkpoint advancing)? ETA to the K-budget target. ## Running it / offloading to a subagent The W&B pull is fast (seconds), unlike the harbor analyzer — you usually do NOT need a subagent. If a run is huge or you are sweeping several, the `analyze-job-history-iris` foreground-and-wait discipline still applies to any slow `gsutil`/log reads, but the W&B query itself is quick. ## Related skills - **monitor-cron-sweep-iris** — the every-3-hours sweep; its **class E** (executor/Levanter training) invokes this skill, just as class A/B invoke `analyze-job-history-iris`. - **analyze-job-history-iris** — the harbor (datagen/eval) analyzer; does NOT apply to training runs. - **rl-job-health-deep-dive** — the GPU-RL equivalent (also no harbor sidecars; uses the finelog).