--- name: dotnet-threads-analysis description: >- Analyze .NET application threading issues using CLI diagnostic tools. Use when investigating thread contention, deadlocks, thread pool starvation, sync-over-async patterns, or frozen/unresponsive applications — on a live process or from a memory dump. Works with dotnet-counters, dotnet-trace, dotnet-dump, and dotnet-pstacks. --- # .NET Thread Analysis Diagnose thread contention, deadlocks, thread pool starvation, and sync-over-async issues in .NET applications using the standard Microsoft diagnostic CLI tools and custom ClrMD-based tools — no code changes to the target application required. ## Prerequisites The following tools must be installed as global .NET CLI tools: ```bash dotnet tool install -g dotnet-dump dotnet tool install -g dotnet-counters dotnet tool install -g dotnet-trace dotnet tool install -g dotnet-pstacks ``` Verify with `dotnet tool list -g`. A .NET runtime (6.0+) is required on the machine; an SDK is not needed to *run* the tools. ## Key Technique: Non-Interactive SOS Commands `dotnet-dump analyze` is interactive by default, which does not work well with agents. Always use the `-c` flag to run SOS commands non-interactively: ```bash dotnet-dump analyze -c "" -c "" -c "exit" ``` Multiple `-c` flags are executed in sequence. Always end with `-c "exit"`. --- **Note** Always start with `threads` and `dotnet-pstacks` for a quick overview of thread state distribution before diving into individual call stacks. On Windows, if WinDbg is installed, use `!locks` and `!runaway` for additional native lock and CPU-time-per-thread details. ## Live Process Investigation Use this workflow when the application is still running and you want to observe threading behavior or detect contention/starvation without taking a full dump. ### Identify the Target ```bash dotnet-dump ps ``` Lists running .NET processes with PID and name. Confirm the target PID. ### Monitor Thread Counters ```bash dotnet-counters monitor -p --counters System.Runtime ``` Key counters to watch: - `threadpool-thread-count` — climbing steadily = possible starvation or sync-over-async pattern - `threadpool-queue-length` — high values = work items waiting for threads - `monitor-lock-contention-count` — spikes = lock contention - `threadpool-completed-items-count` — compare with queue length to gauge throughput Watch counters for at least 30 seconds to distinguish a genuine trend from a transient spike. If `threadpool-thread-count` is **steadily climbing** (not just spiking under load), this is a strong signal of sync-over-async or thread pool starvation. Compare `threadpool-queue-length` against `threadpool-completed-items-count` over time: a growing queue with flat completions confirms the pool cannot keep up. Press `q` to stop. ### Collect an EventPipe Trace ```bash dotnet-trace collect -p --providers Microsoft-Windows-DotNETRuntime ``` Produces a `.nettrace` file for offline analysis. Add `Microsoft-DotNETCore-SampleProfiler` for CPU profiling to identify hot threads. ### Escalate to Full Dump If counters reveal contention or starvation but you need to see exact call stacks and lock ownership, capture a full dump: ```bash dotnet-dump collect -p ``` **Warning:** This briefly freezes the process. Confirm with the user before running on a production system. Then continue with the **Dump-Based Investigation** workflow below. --- ## Dump-Based Investigation Use this workflow when analyzing a `.dmp` file — either provided by the user or captured via `dotnet-dump collect`. ### Parallel Stacks Overview ```bash dotnet-pstacks ``` Merges threads with identical call stacks. Quickly reveals: - Many threads blocked at the same lock frame -> contention - Two groups stuck in opposite lock acquisition -> potential deadlock ### List All Managed Threads ```bash dotnet-dump analyze -c "threads" -c "exit" ``` Shows thread ID, OS ID, state, lock count, and exception info. ### Full Stack Trace for All Threads ```bash dotnet-dump analyze -c "clrstack -all" -c "exit" ``` Look for `Monitor.Enter`, `Monitor.Wait`, `SemaphoreSlim.Wait`, or `ManualResetEventSlim.Wait` frames to identify where threads are blocked. ### Sync Block Table (Lock Ownership) ```bash dotnet-dump analyze -c "syncblk" -c "exit" ``` Shows which thread owns each monitor lock. Columns: - **MonitorHeld**: lock held count - **Owning Thread**: the thread ID that holds the lock - **SyncBlock Owner**: the object being locked on ### Deadlock Detection When the application is frozen and not responding: 1. Run `syncblk` to find which threads own locks 2. Run `clrstack -all` to see where each thread is blocked 3. Cross-reference: if Thread A owns Lock 1 and is blocked waiting for Lock 2, while Thread B owns Lock 2 and is blocked waiting for Lock 1 -> **deadlock** 4. Run `dotnet-pstacks` for visual confirmation of the circular wait pattern **Quick deadlock check sequence:** ```bash dotnet-dump analyze -c "syncblk" -c "threads" -c "clrstack -all" -c "exit" ``` Then run separately: ```bash dotnet-pstacks ``` Report the deadlock cycle clearly: "Thread X holds Lock A (object 0x...) and waits for Lock B (object 0x...); Thread Y holds Lock B and waits for Lock A." ### Inspect Contended Code When a culprit is found (contended lock, deadlock participant, or sync-over-async blocker), inspect the lock object and surrounding code to understand the root cause: ```bash dotnet-dump analyze -c "dumpobj " -c "exit" ``` Use `dumpmt`, `dumpclass`, and `dumpil` to examine the type that owns or acquires the lock. Another strategy is to dump the assembly from the dump with `.writemem` on the corresponding module and decompile the class methods with ilspycmd (to install with `dotnet tool install ilspycmd -g` if needed) to review the actual lock acquisition order, async call chains, or missing `ConfigureAwait(false)` calls. --- ## Diagnosis Patterns | Symptom | Likely Cause | Key Command | |---------|-------------|-------------| | Many threads at same lock frame | Lock contention | `dotnet-pstacks` + `syncblk` | | App frozen, threads waiting on locks | Deadlock (circular dependency) | `syncblk` + `clrstack -all` + `dotnet-pstacks` | | Thread pool count climbing (live) | Sync-over-async / starvation | `dotnet-counters` | | High queue length, low completion rate | Thread pool exhaustion | `dotnet-counters` | | Threads blocked on `SemaphoreSlim.Wait` | Async throttle saturation | `clrstack -all` + `dotnet-pstacks` | | Many threads blocked at `Task.Result` / `.GetAwaiter().GetResult()` | Sync-over-async | `clrstack -all` + `dotnet-pstacks` | ## Investigation Summary — summary markdown file Throughout the investigation, maintain a summary file with the following format `-