--- name: debug-slow-spark-job description: Diagnose slow, expensive, or regressed Apache Spark and PySpark applications by comparing runtime evidence against a healthy run. Use for long stages, stragglers, skew, shuffle, spill, garbage collection, poor parallelism, small files, slow scans, scheduler delay, executor imbalance, and unexplained compute-cost growth. --- # Debug a slow Spark job Find the root cause of the slowdown or cost growth. Compare against a healthy run whenever one exists, and normalize for input size before calling anything a regression. ## Get the evidence ```bash export SPARK_HISTORY_URL="https://history.example.com" python3 scripts/spark_history_api.py slow --app-id > /tmp/spark-slow.json python3 scripts/spark_history_api.py slow --app-id > /tmp/spark-healthy.json ``` Run from this skill directory. If `SPARK_HISTORY_URL` is unset, find the server before asking the user: try `http://localhost:18080`, a running application's UI on `http://localhost:4040`, and the history-server or eventLog settings in the local Spark config; ask only when nothing responds. Authentication comes from `SPARK_HISTORY_AUTHORIZATION`, `SPARK_HISTORY_COOKIE`, or `SPARK_HISTORY_HEADERS_JSON`; never ask for credentials in chat and never disable TLS verification (`--ca-file` for a private CA). ## Collector reference `scripts/spark_history_api.py` subcommands (read-only; each also accepts `--base-url`, `--stage-limit N`, `--task-limit N`): - `applications [--status completed|running] [--limit N]`: list applications and attempt IDs. - `failure --app-id `: `failedJobs`, `failedStages[]` (stage, `taskSummary` quantiles, failed-task samples), `allExecutors`, `environment`, `sqlExecutions`. - `slow --app-id `: `jobs`, `longestStages[]` (stage, `taskSummary` quantiles, slowest-task samples), `allExecutors`, `environment`, `sqlExecutions`. - `sql-list --app-id `: SQL executions with IDs, status, and duration. - `sql --app-id --execution-id `: `sqlExecution` (plan text plus per-node metrics), `jobs`, `longestStages`, `environment`. For anything the profiles omit, call the same REST API directly at `$SPARK_HISTORY_URL/api/v1` with the auth headers above: `/applications/{app}/jobs`, `/stages/{stage}/{attempt}/taskSummary?quantiles=0.05,0.5,0.95`, `/stages/{stage}/{attempt}/taskList?sortBy=-runtime`, `/allexecutors`, `/environment`, `/sql/{execution}?details=true&planDescription=true`, and `/logs` (full event-log zip; large). Slice snapshots rather than reading them whole, e.g. `jq '.longestStages[] | {stage: .stage.stageId, summary: .taskSummary}' /tmp/spark-slow.json`; use the `sql` subcommand when a cause needs the executed plan. Read `spark.master` and deploy mode from `environment` to know where driver and executor logs live. Compare stage timelines between the two runs and start where they first diverge; wall time alone mixes queue time, driver work, execution, and commit. ## Likeliest causes, in order, and where to look 1. **Skew.** In the divergent stage's `taskSummary`, compare max against median run time, input bytes, and shuffle read; the slowest-task samples name the hot partitions and hosts. Check the final plan for AQE skew handling (`skewed=true` on the join or AQEShuffleRead) before proposing salting. 2. **Wrong partition count.** Stage `numTasks` against total cores from `allExecutors`, with `spark.sql.shuffle.partitions` and `spark.executor.cores` from `environment`. Tasks uniformly large and spilling mean too few partitions; tens of thousands of sub-second tasks mean scheduler overhead dominates. 3. **Excess spill.** `memoryBytesSpilled` and `diskBytesSpilled` in `taskSummary`. Spill without skew means operator state outgrew execution memory: reduce state (narrower rows, partial aggregation) or raise partitions before raising memory. 4. **Large shuffles.** Stage `shuffleReadBytes`/`shuffleWriteBytes` dominating its runtime. Map the stage to its `Exchange` in the executed plan and ask whether the shuffle is avoidable: broadcast, pre-aggregation, or already-partitioned data. 5. **Slow shuffle fetch.** `fetchWaitTime` and remote-read bytes in `taskSummary` with low CPU time; dead entries in `allExecutors` mean the data had to be refetched or recomputed. If `environment` shows Celeborn or an external shuffle service, confirm from the driver log it actually served the shuffle before tuning it. 6. **GC pressure.** `jvmGcTime` as a fraction of run time in `taskSummary`, and per-executor `totalGCTime` versus `totalDuration` in `allExecutors`. Check peak memory, cache use, and object-heavy code before resizing heaps. 7. **Python UDF transport.** `BatchEvalPython` or `ArrowEvalPython` nodes in the executed plan and their time relative to rows processed. Prefer native expressions or vectorized UDFs over resource changes. 8. **Retry churn.** `numFailedTasks` on stages and jobs that ultimately succeeded; task samples with `attempt` > 0. Find the flaky cause (one bad host, preemption, timeouts) in the executor log or node events. 9. **Scan overhead.** Scan node metrics in the SQL profile (files read, bytes read, rows output) against stage input metrics: too many small files, missing partition or pushed filters, or slow storage. 10. **Driver and queue time.** Gaps between application start, job `submissionTime`, and first stage `submissionTime`, or between jobs. That time is planning, file listing, queueing, or provisioning: check the driver log for what it was doing, not stage tuning. For a regression, end by naming what changed: code, data volume or distribution, configuration (diff the two snapshots' `environment`), or infrastructure. Adding memory for a skewed partition, adding executors when task count caps parallelism, and caching once-used data are common wrong answers. ## Report State the root cause and confidence, the first divergent stage, the evidence for and against, alternatives you rejected, and the most likely fix. Do not change production settings or launch expensive reruns without approval.