---
name: debug-spark-failure
description: Diagnose failed Apache Spark and PySpark applications from History Server evidence, logs, and cluster-manager state. Use for driver or executor crashes, out-of-memory errors, fetch failures, task exceptions, timeouts, repeated retries, aborted stages, and intermittent production failures.
---

# Debug a Spark failure

Find the root cause: the earliest failure that explains the rest of the chain. Later fetch failures, retries, and executor loss are usually fallout, not cause.

## Get the evidence

```bash
export SPARK_HISTORY_URL="https://history.example.com"
python3 scripts/spark_history_api.py applications --status completed --limit 20
python3 scripts/spark_history_api.py failure --app-id <application-id> > /tmp/spark-failure.json
```

Run from this skill directory. If `SPARK_HISTORY_URL` is unset, find the server before asking the user: try `http://localhost:18080`, a running application's UI on `http://localhost:4040`, and the history-server or eventLog settings in the local Spark config; ask only when nothing responds. Authentication comes from `SPARK_HISTORY_AUTHORIZATION`, `SPARK_HISTORY_COOKIE`, or `SPARK_HISTORY_HEADERS_JSON`; never ask for credentials in chat and never disable TLS verification (`--ca-file` for a private CA).

## Collector reference

`scripts/spark_history_api.py` subcommands (read-only; each also accepts `--base-url`, `--stage-limit N`, `--task-limit N`):

- `applications [--status completed|running] [--limit N]`: list applications and attempt IDs.
- `failure --app-id <id>`: `failedJobs`, `failedStages[]` (stage, `taskSummary` quantiles, failed-task samples), `allExecutors`, `environment`, `sqlExecutions`.
- `slow --app-id <id>`: `jobs`, `longestStages[]` (stage, `taskSummary` quantiles, slowest-task samples), `allExecutors`, `environment`, `sqlExecutions`.
- `sql-list --app-id <id>`: SQL executions with IDs, status, and duration.
- `sql --app-id <id> --execution-id <n>`: `sqlExecution` (plan text plus per-node metrics), `jobs`, `longestStages`, `environment`.

For anything the profiles omit, call the same REST API directly at `$SPARK_HISTORY_URL/api/v1` with the auth headers above: `/applications/{app}/jobs`, `/stages/{stage}/{attempt}/taskSummary?quantiles=0.05,0.5,0.95`, `/stages/{stage}/{attempt}/taskList?sortBy=-runtime`, `/allexecutors`, `/environment`, `/sql/{execution}?details=true&planDescription=true`, and `/logs` (full event-log zip; large).

Failed-task samples carry `errorMessage`, `host`, `executorId`, and partition `index`; executors carry `removeReason`, `failedTasks`, and GC time. Slice snapshots rather than reading them whole, e.g. `jq '.failedStages[].tasks[] | {index, host, executorId, errorMessage}' /tmp/spark-failure.json`. The History Server has no driver logs or container termination reasons: read `spark.master` and `spark.submit.deployMode` from `environment` to locate them, then pull the driver log around the first exception, the first failing executor's log (`kubectl logs --previous`, `yarn logs -applicationId`, or the platform's log store), and the cluster manager's reason for any lost container. A driver crash can leave no failed Spark job at all.

## Likeliest causes, in order, and where to look

1. **Executor OOM.** `allExecutors` entries with `removeReason` mentioning OOM or heap errors in the executor log; the decisive record is the cluster manager's exit code (137 / `OOMKilled` from `kubectl describe pod` or events, YARN container exit status). Then read the failed stage's `taskSummary` input and shuffle quantiles: one oversized partition or join state is the usual cause, not globally low memory.
2. **Skew surfacing as failure.** In the failed stage's `taskSummary`, compare the max against the median for run time, input bytes, and shuffle read. If max is many times median, fix the key distribution, not the resources.
3. **Fetch failures.** The failed tasks' `errorMessage` names the executor and shuffle block that could not be fetched: look that executor up in `allExecutors` for its `removeReason` and time, then diagnose its earlier death from its own log. If `environment` shows Celeborn or an external shuffle service, confirm in the driver log that it actually served the affected shuffle; clients can fall back to built-in shuffle.
4. **Flaky infrastructure.** Group the failed task samples by `host` and `executorId`. Failures clustered on one host point at infrastructure: check that node's events (`kubectl get events`, cluster-manager node state), spot preemption, and disk or network errors in the executor log. Tasks that later succeeded on retry confirm flakiness rather than bad data.
5. **Deterministic data or code failure.** The same exception at the same partition `index` across attempts and executors in the task samples: corrupt input, a pathological record, or a user-code bug. The partition identity is your reproduction.
6. **Python worker death.** Task `errorMessage` reports the Python worker exiting; the traceback is in that executor's stderr, and `spark.executor.pyspark.memory` in `environment` bounds worker memory.
7. **Driver failure.** The app died with no failed jobs: read the end of the driver log. Large collect, broadcast, or result sizes and very high partition counts (task-metadata bloat) are common; a GC-stalled driver also produces heartbeat storms visible in executor logs.
8. **Timeouts.** Heartbeat and RPC timeout messages are almost always secondary. Cross-check `allExecutors` GC time and the stage's task distribution to find what was actually slow instead of raising the timeout.
9. **Commit or write failure.** Committer and destination errors are in the driver log and failed task messages: look for concurrent writers, retried stages double-committing, permissions, and storage throttling.

Before concluding, merge the driver log, executor logs, and platform events into one timeline and take the earliest error that explains the rest. Configuration proves intent, not behavior; prefer runtime evidence.

## Report

State the root cause and confidence, the failure chain from it, the evidence for and against, the smallest fix or reproduction, and what evidence was missing. Say "confirmed" only when logs or a reproduction establish causality; otherwise "leading hypothesis". Do not change production configuration or rerun expensive jobs without approval.