--- name: hyperpod-cluster-debugger description: Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only. metadata: version: "0.0.1" --- # HyperPod Cluster Debugger **Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). **Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review). `scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/.md §
`. | Reference | Open when | | ------------------------------------------------------------------------- | ------------------------------------------------------------------- | | [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L) | | [cluster-operations.md](references/cluster-operations.md) | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) | | [cloudformation-errors.md](references/cloudformation-errors.md) | § H needs the full per-resource CFN error catalog | | [capacity-planning.md](references/capacity-planning.md) | § B or `--validate` flags capacity / subnet sizing | | [lifecycle-scripts.md](references/lifecycle-scripts.md) | § C points at a specific lifecycle failure | | [iam-permissions.md](references/iam-permissions.md) | Full IAM policy for the diagnostic | --- ## Workflow 1. Collect HyperPod cluster name (not EKS name), region, exact error string. 2. Run `scripts/diagnose-cluster.sh` (or `--validate` for pre-create). 3. For every `[FAIL]` line, `Read` the referenced section. 4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval. 5. Re-run the diagnostic to confirm. --- ## Step 1: Run diagnostics ```bash # Diagnose an existing cluster: bash scripts/diagnose-cluster.sh --cluster --region # Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints, # optionally S3 lifecycle scripts and per-AZ capacity: bash scripts/diagnose-cluster.sh --validate --region \ --sg-ids --subnet-ids [--iam-role ] \ [--s3-uri s3:///path/] [--instance-type ml.p5.48xlarge] ``` Pass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time). Tags: `[PASS]` · `[FAIL]` (counted, has `→ references/...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational. --- ## Step 2: Match signal → section **Error messages / events:** | Signal | Section | | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | | `"EFA health checks did not run successfully"` (public-doc verbatim signal) | **[A: EFA Health Checks](#a-efa-health-checks)** | | Insufficient-capacity or AZ-mismatch failure at creation | **[B: Capacity & AZ](#b-capacity--az)** | | Lifecycle-script failure or timeout during provisioning | **[C: Lifecycle Scripts](#c-lifecycle-scripts)** | | kubectl auth error (server asks for credentials / no API group list) | **[D: EKS Access](#d-eks-access--kubectl)** | | `InService` but not all instances visible | **[E: Cluster Provisioning](#e-cluster-provisioning)** | | `"Target is not connected"` / SSM errors | **[F: SSM Connectivity](#f-ssm-connectivity)** | | Node replacement not happening / `batch-replace` not working | **[G: Node Replacement](#g-node-replacement)** | | `"Embedded stack failed"` / any CloudFormation error | **[H: CloudFormation Errors](#h-cloudformation-errors)** | | `UpdateClusterSoftware` failed or cluster in post-maintenance rollback state | **[J: AMI & Cluster Updates](#j-ami--cluster-updates)** | | Dangling / orphaned nodes in EKS vs `list-cluster-nodes` | **[K: Dangling Nodes & Cleanup](#k-dangling-nodes--cleanup)** | | Cluster Autoscaler breaks after HyperPod attached | **[L: Autoscaler Compatibility](#l-autoscaler-compatibility)** | | Slow I/O, FSx throughput saturated | [cluster-operations.md § 9](references/cluster-operations.md) | | Slurm node name → instance ID lookup | **[I: Utilities](#i-utilities)** | --- ## A: EFA Health Checks SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: [cluster-diagnostics-detail.md § A](references/cluster-diagnostics-detail.md#a-efa-health-checks). ## B: Capacity & AZ Instance type unavailable in the requested AZ. Verify with `describe-instance-type-offerings`, then change AZ, use Flexible Training Plans, or request ODCR. Full: [§ B](references/cluster-diagnostics-detail.md#b-capacity--az) · strategy: [capacity-planning.md](references/capacity-planning.md). ## C: Lifecycle Scripts Script failed or timed out during provisioning. Read CloudWatch under `/aws/sagemaker/Clusters//` — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: [§ C](references/cluster-diagnostics-detail.md#c-lifecycle-scripts) · layout: [lifecycle-scripts.md](references/lifecycle-scripts.md). ## D: EKS Access / kubectl IAM identity not in EKS access entries. Verify with `sts get-caller-identity`, create an access entry with admin policy, update kubeconfig. Full: [§ D](references/cluster-diagnostics-detail.md#d-eks-access--kubectl). ## E: Cluster Provisioning `InService` without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck `Creating`/`Updating`/`Deleting`: check CFN nested stacks (§ H), IAM, capacity, events; if stuck `Deleting` check VPC ENI dependencies. Full: [§ E](references/cluster-diagnostics-detail.md#e-cluster-provisioning). ## F: SSM Connectivity `Target is not connected`: use `sagemaker-cluster:_-` format (not raw EC2 ID), install session-manager-plugin, confirm node `Running`. Check IAM + VPC endpoints on timeouts. Full: [§ F](references/cluster-diagnostics-detail.md#f-ssm-connectivity). ## G: Node Replacement Auto-repair: confirm `NodeRecovery=Automatic`, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via `UpdateClusterSoftware` at least once and cannot target a Slurm controller node. Full: [§ G](references/cluster-diagnostics-detail.md#g-node-replacement). ## H: CloudFormation Errors `Embedded stack failed` hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: `describe-stack-events --query 'StackEvents[?ResourceStatus==\`CREATE_FAILED\`]'`. Also covers SLR creation failures and permission-boundary denials. Full: [§ H](references/cluster-diagnostics-detail.md#h-cloudformation-errors) · catalog: [cloudformation-errors.md](references/cloudformation-errors.md). ## I: Utilities Map Slurm node names (`ip-10-x-y-z`) to HyperPod instance IDs via `list-cluster-nodes` or on-node `/opt/ml/config/resource_config.json`. Full: [§ I](references/cluster-diagnostics-detail.md#i-utilities). ## J: AMI & Cluster Updates `UpdateClusterSoftware` fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: [§ J](references/cluster-diagnostics-detail.md#j-ami--cluster-updates). ## K: Dangling Nodes & Cleanup Nodes in `kubectl get nodes` but not in `list-cluster-nodes` (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: [§ K](references/cluster-diagnostics-detail.md#k-dangling-nodes--cleanup). ## L: Autoscaler Compatibility Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: [§ L](references/cluster-diagnostics-detail.md#l-autoscaler-compatibility). --- ## Prerequisites - `aws` CLI v2.13+ authenticated to the cluster's account - `jq`, `python3`, `bash` 4.2+ - `kubectl` authenticated to the EKS cluster (EKS checks skipped if absent) - `session-manager-plugin` (Slurm controller health checks only) IAM policy: [references/iam-permissions.md](references/iam-permissions.md). ## Defaults - **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`. - **Mode** — `--cluster ` (diagnose) or `--validate` (pre-create). - **Event window** — up to 500 most recent events (5 × 100, paginated). - **Colors** — auto-disabled on non-TTY; `--no-color` to force off. ## Error handling | Failure | Script | Tell the customer | | --------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- | | `aws sts get-caller-identity` fails | Exit 1 | "Fix AWS credentials and rerun." | | Cluster not found | Exit 1 after listing region's clusters | "Confirm HyperPod cluster name (not EKS) and region." | | `sagemaker:*` / `ec2:*` / `eks:*` / `logs:*` denied | Warn, add `Missing IAM permission for `, continue | "Grant the listed IAM action and rerun." | | `kubectl` absent or unauthenticated | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | "Install/authenticate kubectl." | | `session-manager-plugin` absent (Slurm) | Skip Slurm controller probe | "Install session-manager-plugin." | | SSM throttled / times out (180s) | Retry with backoff; warn and continue if still failing | "Rerun later — script is idempotent." | | CloudWatch log group not found | Skip CloudWatch check | "CloudWatch not configured on this cluster." | Exit codes: `0` no critical failures · `1` one or more critical failures (cluster not found, fatal prerequisite missing, or any `[FAIL]` in diagnose or `--validate` mode). `[WARN]` lines do not affect the exit code. ## Skill delegation | Need | Use | | ------------------------------- | -------------------------- | | Shell on nodes | `hyperpod-ssm` | | Version comparison across nodes | `hyperpod-version-checker` | ## Escalate to AWS Support Escalate when: 1. EFA health checks fail despite correct SG rules. 2. Capacity errors persist despite a valid Flexible Training Plan / ODCR. 3. Node replacement fails repeatedly without clear events / log signal. 4. Cluster stuck in a non-terminal state (`Creating`, `Updating`, or a post-maintenance rollback state) for an extended period. 5. CloudFormation root-cause is an internal service error. ### Before opening the case Run these commands and attach the output. Goal: AWS Support has everything at case open. ```bash # 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups) aws sagemaker describe-cluster --cluster-name --region # 2. Full cluster-level diagnostic bundle bash scripts/diagnose-cluster.sh --cluster --region > diag.txt # 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill) # See skills/hyperpod-issue-report/SKILL.md for the exact invocation. ``` ### Include in the case - Cluster name + ARN (or `ClusterId` suffix) and AWS region - `ClusterStatus` + `FailureMessage` from `describe-cluster` - Timestamp window (UTC start / end) of the failure - Exact error strings observed (copy verbatim from events / logs / console) - Affected instance IDs / `NodeLogicalId`s / instance group names - `diag.txt` from step 2 above - S3 URI of the `hyperpod-issue-report` bundle from step 3