--- name: debug-openshell-cluster description: Debug why an OpenShell gateway deployment is unhealthy, unreachable, or unable to create sandboxes. Use for gateway health failures, Docker/Podman runtime issues, Helm failures, Kubernetes scheduling, TLS or auth, gateway interceptors, supervisor middleware startup or runtime failures, external compute-driver sockets, VM drivers, or sandbox startup. Trigger keywords - debug gateway, gateway failing, deployment failing, helm install failing, cluster health, gateway health, gateway not starting, health check failed, sandbox pending, docker driver, podman driver, kubernetes driver, external driver, compute driver socket, gateway interceptor, supervisor middleware, middleware failed, vm driver. --- # Debug OpenShell Gateway Deployment Diagnose a gateway and its selected compute platform. Do not assume OpenShell provisions Kubernetes or runs a k3s container. OpenShell targets a reachable gateway endpoint backed by Docker, Podman, Kubernetes, the experimental VM driver, or an operator-managed out-of-tree compute driver. Use `openshell` first to identify the active endpoint. Then use the platform tools that match the gateway's compute driver: `docker`, `podman`, `kubectl`/`helm`, or VM driver logs. ## Overview The target deployment flow is: 1. Operator starts or deploys the gateway with system packages, systemd, Helm, or a development task. The CLI does not start, stop, or destroy gateway services. 2. Operator configures the compute driver. 3. Operator provides the CLI and supervisor authentication material required by the deployment mode: edge or OIDC user auth, optional CLI mTLS, and gateway-minted sandbox JWTs. 4. The CLI registers a reachable gateway endpoint with `openshell gateway add`. 5. The gateway creates sandboxes through the selected compute driver. For local evaluation only, TLS may be disabled and the gateway can be reached through `http://127.0.0.1:`. ## Prerequisites - The `openshell` CLI must be available for endpoint checks. - Know the active gateway name and endpoint, or be able to inspect local gateway metadata. - Know the compute platform: Docker, Podman, Kubernetes, VM, or an out-of-tree driver. - For Kubernetes: `kubectl` must target the cluster that hosts OpenShell and Helm version 3 or later must be available. - For Docker or Podman: the runtime socket must be reachable from the gateway host. ## Workflow Run diagnostics in order and stop once the root cause is clear. ### Step 1: Check CLI Reachability ```bash openshell gateway list --output json openshell gateway info openshell status ``` For a one-off endpoint check that bypasses stored gateway selection and metadata: ```bash openshell --gateway-endpoint status ``` Common findings: - `No active gateway`: register one with `openshell gateway add `. - Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active. - TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener. - `Unauthenticated` from an edge or OIDC gateway: refresh stored credentials with `openshell gateway login [name]`, then retry. Use `gateway logout` only when intentionally clearing local credentials. - A direct development endpoint with a private or self-signed certificate can be isolated with `--gateway-endpoint --gateway-insecure`; do not persist or recommend insecure verification for shared gateways. ### Step 2: Identify the Compute Platform Use gateway metadata, deployment values, or the user's setup notes to identify the driver. | Platform | Primary checks | |---|---| | Docker | Gateway process logs, Docker daemon health, sandbox containers, image pulls. | | Podman | Podman socket, rootless networking, sandbox containers, image pulls. | | Kubernetes | Helm release, gateway workload, service, secrets, sandbox pods, events. | | VM | VM driver logs, rootfs availability, host virtualization support. | | Extension | External driver process, Unix socket ownership/mode, configured driver name, capability handshake, gateway logs. | ### Step 3: Check Gateway Startup Dependencies Before debugging the compute platform, inspect gateway logs for failures in dependencies initialized before the listener becomes ready. For out-of-tree compute drivers, confirm the custom driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts: ```bash rg -n 'compute_drivers|socket_path' /etc/openshell/gateway.toml stat /run/openshell/.sock journalctl -u --no-pager --lines=200 journalctl -u openshell-gateway --no-pager --lines=200 ``` The custom driver name must not be a reserved built-in name (`docker`, `podman`, `kubernetes`, or `vm`). The socket must be accessible only to the intended gateway identity. Check gateway logs for connection errors, `GetCapabilities` failures, or an unexpected advertised driver name. The gateway does not create or supervise out-of-tree driver processes or sockets. For configured gateway interceptors, inspect `[[openshell.gateway.interceptors]]`, their Unix or network endpoints, and gateway startup logs: ```bash rg -n 'interceptors|provider_profile_sources|grpc_endpoint|binding_policy|failure_policy' /etc/openshell/gateway.toml stat /run/openshell/interceptors/.sock journalctl -u --no-pager --lines=200 journalctl -u openshell-gateway --no-pager --lines=200 ``` The gateway calls each interceptor's `Describe` RPC and validates its manifest at startup. Check for unreachable endpoints, invalid RPC/phase bindings, strict `allowlist` or `exact` mismatches, and `post_commit` bindings that resolve to `fail_closed`. If `provider_profile_sources` names an interceptor, that interceptor must advertise provider-profile capability and return a valid, duplicate-free catalog. A selected interceptor-only source is authoritative; include `builtin` or `user` sources explicitly when composition is intended. For operator-run supervisor middleware, inspect `[[openshell.supervisor.middleware]]`, service reachability, and both gateway and supervisor logs: ```bash rg -n 'supervisor|middleware|grpc_endpoint|max_body_bytes|timeout' /etc/openshell/gateway.toml journalctl -u --no-pager --lines=200 journalctl -u openshell-gateway --no-pager --lines=200 openshell logs --tail --source sandbox ``` The middleware service must start before the gateway and be reachable from both the gateway and sandbox supervisors. Gateway startup fails if `Describe` is unavailable, a manifest exposes duplicate `HttpRequest/pre_credentials` bindings, the registration claims the reserved `openshell/` namespace, or body and timeout limits are invalid. Changing a registration requires a gateway restart. A policy update can also fail before persistence if the selected implementation rejects its `network_middlewares` config. At request time, distinguish an explicit `middleware_denied` result from `middleware_failed`. A denial is always enforced. A failure follows the policy-local `on_error`: `fail_closed` blocks the request, while `fail_open` bypasses only that stage and emits a detection finding. If a running supervisor cannot install a new registry, it preserves its last-known-good generation and emits a configuration failure event. ### Step 4: Check Docker-Backed Gateways ```bash docker info docker ps --filter name=openshell docker logs --tail=200 docker run --rm --entrypoint /openshell-sandbox "${OPENSHELL_DOCKER_SUPERVISOR_IMAGE:-ghcr.io/nvidia/openshell/supervisor:latest}" --version openshell status ``` For Docker GPU failures, check CDI support and NVIDIA CDI discovery separately: ```bash docker info --format '{{json .CDISpecDirs}}' docker info --format '{{json .DiscoveredDevices}}' for dir in /etc/cdi /var/run/cdi; do if [ -d "$dir" ]; then find "$dir" -maxdepth 1 -type f \( -name '*.yaml' -o -name '*.json' \) -print else echo "$dir missing" fi done systemctl is-enabled nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true systemctl is-active nvidia-cdi-refresh.service nvidia-cdi-refresh.path || true systemctl status nvidia-cdi-refresh.service nvidia-cdi-refresh.path --no-pager --lines=50 journalctl -u nvidia-cdi-refresh.service --no-pager --lines=100 ``` When the NVIDIA Container Toolkit CDI refresh units are not enabled or no NVIDIA CDI spec has been generated, enable them and trigger a refresh: ```bash sudo systemctl enable --now nvidia-cdi-refresh.path sudo systemctl enable --now nvidia-cdi-refresh.service sudo systemctl restart nvidia-cdi-refresh.service docker info --format '{{json .DiscoveredDevices}}' ``` Common findings: - Docker daemon unavailable: start Docker Desktop or Docker Engine. - Gateway process stopped: inspect exit status and logs. - Sandbox image missing or pull denied: verify image reference and registry credentials. - Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected. - Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`. - Sandbox never registers: check gateway logs and supervisor callback endpoint. - Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`. - `mise run e2e:docker:gpu` fails with `docker info --format json did not report any discovered NVIDIA CDI GPU devices`: Docker may report `CDISpecDirs` while still having no generated NVIDIA CDI specs. Verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. NVIDIA recommends enabling the path and service units, and restarting `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs. If specs are generated but Docker still reports no discovered devices, restart Docker or reload the daemon and re-check `docker info`. For source checkout development, restart the local gateway with: ```bash mise run gateway:docker ``` ### Step 5: Check Podman-Backed Gateways ```bash podman info podman ps --filter name=openshell podman logs --tail=200 openshell status ``` Common findings: - Podman socket unavailable: start or expose the user socket. - Rootless networking unavailable: inspect Podman network configuration. - Sandbox image missing or pull denied: verify image reference and registry credentials. - Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected. - Supervisor cannot call back: check callback endpoint and gateway logs. ### Step 6: Check Kubernetes Helm Gateways ```bash helm -n openshell status openshell helm -n openshell get values openshell kubectl -n openshell get deployment,statefulset,pod,svc,pvc kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 kubectl -n openshell rollout status deployment/openshell kubectl -n openshell rollout status statefulset/openshell ``` Use the log and rollout commands for the workload kind that exists in the release. Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures. For HA or PostgreSQL-backed installs, also check the external database Secret referenced by `server.externalDbSecret` and the PostgreSQL workload if the test or operator deployed one in-cluster: ```bash kubectl -n openshell get secret openshell-ha-pg -o yaml kubectl -n openshell get deployment,service,pod -l app.kubernetes.io/name=openshell-e2e-postgres kubectl -n openshell logs deployment/openshell-e2e-postgres --tail=200 ``` Check required Helm deployment secrets: ```bash kubectl -n openshell get secret \ openshell-server-tls \ openshell-server-client-ca \ openshell-client-tls \ openshell-jwt-keys ``` In cert-manager installs, `certManager.enabled=true` makes cert-manager own TLS generation. The Helm chart should still render the `openshell-certgen` pre-install/pre-upgrade hook in JWT-only mode to create `openshell-jwt-keys`, even if `pkiInitJob.enabled` remains true. If the gateway pod is pending with `MountVolume.SetUp failed for volume "sandbox-jwt"` and `openshell-jwt-keys` is absent, inspect the rendered `templates/certgen.yaml` output and the hook Job logs; cert-manager creates TLS Secrets but does not create the sandbox JWT signing Secret. If the gateway exits with `failed to read sandbox JWT signing key from /etc/openshell-jwt/signing.pem`, verify that `openshell-jwt-keys` contains `signing.pem`, `public.pem`, and `kid`, and that the gateway workload mounts the `sandbox-jwt` secret at `/etc/openshell-jwt`. The sandbox JWT mount is required even when local Helm values disable TLS. If `server.providerTokenGrants.spiffe.enabled=true`, the gateway should still render `[openshell.gateway.gateway_jwt]` and mount the `sandbox-jwt` Secret. SPIRE is used only by sandbox pods for dynamic provider token grants. Verify that SPIRE is installed, the CSI driver is available, and the Kubernetes driver config includes `provider_spiffe_workload_api_socket_path`: ```bash helm -n openshell get values openshell | grep -E 'providerTokenGrants|workloadApiSocketPath' kubectl get pods -A | grep -E 'spire|spiffe' kubectl -n openshell get configmap openshell-config -o yaml | grep provider_spiffe_workload_api_socket_path ``` Sandbox pods using provider token grants should have an `openshell.io/sandbox-id` annotation, an `openshell.ai/managed-by=openshell` label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET`, plus both the projected `openshell-sa-token` volume and the `spiffe-workload-api` CSI volume. Check the image references currently used by the gateway deployment: ```bash kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload' ``` The gateway image built from `deploy/docker/Dockerfile.gateway` and the scratch supervisor image built from `deploy/docker/Dockerfile.supervisor` should use the same build tag in branch and E2E deploys. A stale supervisor image can make sandbox behavior lag behind gateway policy or proto changes. For local/external pull mode (the default local path via `mise run cluster`), local images are tagged to the configured local registry base, pushed to that registry, and pulled by k3s via the `registries.yaml` mirror endpoint. The `cluster` task pushes prebuilt local tags (`openshell/*:dev`, falling back to `localhost:5000/openshell/*:dev` or `127.0.0.1:5000/openshell/*:dev`). Gateway image builds stage a partial Rust workspace from `deploy/docker/Dockerfile.images`. If cargo fails with a missing manifest under `/build/crates/...`, or an imported symbol exists locally but is missing in the image build, verify that every current gateway dependency crate, including `openshell-driver-docker`, `openshell-driver-kubernetes`, and `openshell-ocsf`, is copied into the staged workspace there. For plaintext local evaluation, confirm the chart has: ```bash helm -n openshell get values openshell | grep -E 'disableTls|grpcEndpoint' ``` Expected shape: ```yaml server: disableTls: true grpcEndpoint: http://openshell.openshell.svc.cluster.local:8080 ``` Check service exposure: ```bash kubectl -n openshell get svc openshell -o wide kubectl -n openshell get endpoints openshell ``` For local port-forward testing: ```bash kubectl -n openshell port-forward svc/openshell 8080:8080 openshell gateway add http://127.0.0.1:8080 --local --name local openshell status ``` If the gateway is healthy but sandbox creation fails: ```bash kubectl -n openshell get pods kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50 kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 ``` Check the configured sandbox namespace: ```bash helm -n openshell get values openshell | grep sandboxNamespace ``` Then inspect sandbox resources in that namespace. Check the configured sandbox service account when TokenReview bootstrap or sandbox registration fails. Helm creates a dedicated sandbox service account by default and writes it to `[openshell.drivers.kubernetes].service_account_name`; the gateway rejects projected tokens from other service accounts. ```bash helm -n openshell get values openshell | grep -A3 sandboxServiceAccount kubectl -n get serviceaccount openshell-sandbox kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' kubectl -n get sandbox -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' ``` If `topology = "sidecar"` is rendered under `[openshell.drivers.kubernetes]`, sandbox pods should have an `openshell-network-init` init container running `--mode=network-init`, an `agent` container running `openshell-sandbox --mode=process`, and an `openshell-supervisor-network` container running `--mode=network`. The init container owns nftables setup and should be the only sidecar topology container with `NET_ADMIN`. It also needs `CHOWN`/`FOWNER` to hand shared emptyDir state to the effective sidecar UID. The default binary-aware network sidecar runs as UID 0 with primary GID `sandbox_gid` and adds `SYS_PTRACE` plus `DAC_READ_SEARCH`. When `process_binary_aware_network_policy = false`, it runs as the configured non-root `proxy_uid` without those inspection capabilities. The pod `fsGroup` is set to `sandbox_gid` in both modes. In sidecar topology only the network sidecar should mount the gateway bootstrap credentials (`openshell-sa-token` and `openshell-client-tls`). The process container should not receive `OPENSHELL_ENDPOINT`, gateway TLS env vars, the sandbox token file, or those credential mounts. Instead, the network sidecar serves policy and provider environment state over the Unix control socket from `OPENSHELL_SIDECAR_CONTROL_SOCKET` (`/run/openshell-sidecar/control.sock` by default). The process supervisor must be the first and only client. After validating its peer UID, GID, and PID, the sidecar unlinks the listener. If the connection later closes, the network sidecar exits non-zero so Kubernetes can restart it with a fresh listener. If the process supervisor fails before launching the workload, inspect both containers for control-socket bind, connect, bootstrap, or update errors. If new SSH/exec sessions do not pick up refreshed provider environment, inspect the network sidecar settings-poll logs and the process container logs for provider environment update handling; the process container should consume newer provider-env revisions without receiving gateway credentials. The process container reports the workload entrypoint PID over the same control socket, and the network sidecar uses that PID for binary-scoped policy decisions through `/proc`. If rules with `policy.binaries` are unexpectedly denied, inspect the sidecar control logs and confirm the pod has `shareProcessNamespace: true`. The shared state directory should preserve `sandbox_gid` inheritance (`02775`). Sidecar SSH uses the Linux abstract socket `@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before bridging gateway relay requests. No `ssh.sock` file should appear in the shared state directory. Inspect all three when sandbox registration or egress enforcement fails: ```bash kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*=' kubectl -n get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{" "}{.command}{"\n"}{end}' kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.command}{"\n"}{end}' kubectl -n logs -c openshell-network-init --tail=200 kubectl -n logs -c openshell-supervisor-network --tail=200 kubectl -n logs -c agent --tail=200 ``` ### Step 7: Check VM-Backed Gateways Use the VM driver logs and host diagnostics available in the user's environment. Verify: - The VM driver process is running and reachable by the gateway. - The runtime rootfs exists and matches the expected architecture. - Host virtualization support is enabled. - The sandbox supervisor can establish its callback connection to the gateway. Then run: ```bash openshell status openshell logs ``` ## Common Failure Patterns | Symptom | Likely cause | Check | |---|---|---| | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | | Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` | | Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` | | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways//mtls/` | | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login `, gateway auth logs | | Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy | | Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `builtin` and `user` when intended | | Gateway fails after registering supervisor middleware | Service unavailable, invalid manifest, duplicate binding, reserved name, or invalid body/timeout limit | Middleware service and gateway logs; `[[openshell.supervisor.middleware]]`; `Describe` response | | Policy update rejects `network_middlewares` | Unknown middleware name, implementation-owned config invalid, duplicate order, broad/invalid host selector, or fail-closed coverage of `tls: skip` | Policy error, gateway logs, middleware `ValidateConfig`, selector and order fields | | HTTP request returns `middleware_failed` or `middleware_denied` | Selected stage failed or explicitly denied the admitted request | Sandbox OCSF logs; policy-local middleware config; service availability; `on_error` | | Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or configured with a reserved/mismatched name | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs | | Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials | | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | ## Reporting When handing results back to the user, include: - Active gateway endpoint and auth mode. - Compute platform and driver. - Gateway process or workload status. - Recent gateway log summary. - Missing or malformed TLS, OIDC/mTLS, or sandbox JWT material. - Service exposure status. - Sandbox workload status. - The exact command that failed and the shortest fix.