--- name: kernel-vuln-analyzer description: > Analyze Linux kernel vulnerabilities from KASAN/UBSAN/BUG crash logs or CVE descriptions. Performs full root cause analysis, exploitability assessment, patch development, and verification. Use this skill whenever the user provides a kernel crash log, KASAN report, kernel panic trace, syzbot report, or asks to analyze/patch a kernel vulnerability. Also trigger when the user mentions kernel CVEs, kernel exploit analysis, kernel bug triage, or wants to understand if a kernel bug is exploitable. Even if the user just pastes a raw stack trace from dmesg, this skill applies. --- # Kernel Vulnerability Analyzer A comprehensive skill for analyzing Linux kernel vulnerabilities — from crash log triage through root cause analysis, exploitability assessment, patch development, and verified fix delivery. This skill is designed around a **hive-mode subagent architecture**: break the analysis into parallel workstreams, plan before executing, and coordinate results across agents. ## Core Workflow Overview The analysis follows seven phases. Each phase builds on the previous, but many sub-tasks within a phase can run in parallel via subagents. ``` Phase 1: Triage & Planning ──→ Phase 2: Source Acquisition │ │ │ [1.0: Source Version Check] │ [GATE: Source Version Verified] ↓ ↓ Phase 3: Root Cause Analysis ←── Phase 4: Dynamic Analysis (QEMU+GDB) ↓ Phase 5: Exploitability Assessment ↓ Phase 6: Patch Development & Verification │ │ [6.1: Source Version Re-check] │ [GATE: QEMU Evidence Required] ↓ Phase 7: Report Generation & Artifact Packaging ``` --- ## Phase 1: Triage & Planning **Goal**: Understand what we're dealing with and plan the analysis strategy. Before writing a single line of analysis, enter Plan mode and create a structured plan. This is non-negotiable — kernel bugs are complex and a wrong turn wastes significant time. ### 1.0 Verify Source Tree Is Current (FIRST — Before Anything Else) **This is a BLOCKING prerequisite.** Before parsing the bug report, before reading source code, before planning — verify the kernel source tree is up-to-date. This takes 5 seconds and prevents all downstream version mismatch issues. ```bash # Step 1: Fetch latest from remote cd /path/to/kernel-src git fetch origin --tags # Step 2: Check how far behind we are BEHIND=$(git log --oneline HEAD..origin/master | wc -l) LATEST_TAG=$(git describe --tags --abbrev=0 origin/master 2>/dev/null || echo "unknown") CURRENT=$(git describe --tags HEAD 2>/dev/null || echo "unknown") echo "Current: $CURRENT | Latest upstream: $LATEST_TAG | Commits behind: $BEHIND" # Step 3: If behind → update to latest tag if [ "$BEHIND" -gt 0 ]; then echo "WARNING: Local tree is $BEHIND commits behind upstream ($LATEST_TAG)" echo "Updating to latest tag..." git checkout "$LATEST_TAG" fi # Step 4: Record the base for the report echo "Analysis base: $(git describe --tags HEAD)" ``` **DO NOT proceed to Phase 1.1 until the source tree is on the latest upstream tag.** If `git fetch` fails (no network, wrong remote), warn the user and document that the analysis is based on a potentially stale tree. But ALWAYS attempt the fetch first. ### 1.1 Parse the Input The user may provide: - A raw KASAN/UBSAN/BUG/panic log (most common) - A syzbot report URL or crash description - A CVE identifier - A verbal description of a kernel bug - A PoC (C code, syzkaller repro, etc.) **For crash logs**, extract these key signals: - **Bug type**: KASAN (UAF, OOB-read, OOB-write, double-free), UBSAN, BUG(), WARNING, NULL ptr deref, GPF, etc. - **Faulting address and access type**: Read/Write, address pattern (NULL page, kernel text, slab, etc.) - **Call stack**: The full decoded call trace — this is the most important piece - **Slab cache name**: e.g., `kmalloc-256`, `skbuff_head_cache`, `task_struct` — hints at the object type - **Allocated/Freed stacks**: KASAN often shows where the object was allocated and freed - **Kernel version and config**: What kernel is this running? What configs are relevant? - **Subsystem**: Derived from the call stack — is this networking (net/), filesystem (fs/), drivers, etc.? ### 1.1.1 Decode the Stack Trace (Required) If the crash log contains raw addresses (not resolved to source lines), decode it immediately: ```bash ./scripts/decode_stacktrace.sh vmlinux < crash.log > decoded_crash.log ``` If `vmlinux` is not available, use `addr2line` on individual addresses: ```bash addr2line -e vmlinux -fip 0xffffffff81234567 ``` The **decoded backtrace** is a critical artifact — it is needed for: 1. The root cause analysis (Phase 3) — source file:line references 2. The commit message (Phase 6) — upstream convention requires the decoded trace in the commit body 3. The final report (Phase 7) — annotated trace with file:line Save the decoded trace to `logs/decoded_crash.log` in the report directory. Keep both the raw and decoded versions — raw for reproduction, decoded for analysis. Read `references/crash-log-analysis.md` for detailed patterns and parsing guidance. ### 1.2 Acquire the PoC If the user provides a crash log but no PoC, you need to obtain or create one. **From syzbot**: Read `references/syzbot-workflow.md` for details. ```bash # Download C reproducer from syzbot curl -sL '' -o poc.c gcc -o poc -static -lpthread poc.c # always static-link for QEMU rootfs ``` **From CVE databases**: Search for public PoCs on GitHub, Exploit-DB, or the CVE references. **Write from scratch**: If no PoC exists, write a minimal trigger based on the crash call trace: 1. Identify the syscall entry point from the bottom of the stack trace 2. Set up required preconditions (namespaces, sysctl, devices) 3. Issue the triggering syscall sequence 4. For race conditions: use pthreads to run concurrent paths **PoC validation checklist**: - [ ] Compiles with `gcc -static` (needed for minimal QEMU rootfs) - [ ] Does it need root? Network? Specific sysctl? - [ ] Does it need `unshare(CLONE_NEWUSER|CLONE_NEWNET)` for namespaces? - [ ] Is the crash reliable or does it need loop/stress testing? ### 1.3 Identify the Kernel Subsystem and Source Tree **Do NOT guess the tree from the top-level directory name alone.** Many subsystems have independent maintainer trees even though their code lives under a shared parent directory. The canonical source of truth is `scripts/get_maintainer.pl` and the `MAINTAINERS` file. **Step 1: Run `get_maintainer.pl` on the affected file** ```bash ./scripts/get_maintainer.pl --scm --web # The "SCM:" line tells you the correct git tree # The "W:" line tells you the web page / mailing list ``` **Step 2: Cross-reference with the subsystem tree mapping** Some paths are deceptive — always match **most-specific path first**: | Source path | Subsystem | Git tree (fixes) | Prefix | |---|---|---|---| | `net/bluetooth/` | Bluetooth | `bluetooth/bluetooth.git` | `PATCH bluetooth` | | `net/wireless/`, `drivers/net/wireless/` | WiFi | `wireless/wifi.git` | `PATCH wifi` | | `net/mac80211/` | WiFi (mac80211) | `wireless/wifi.git` | `PATCH wifi` | | `net/netfilter/`, `net/ipv4/netfilter/` | Netfilter | `netfilter/nf.git` | `PATCH nf` | | `net/bridge/` | Bridge | `netdev/net.git` | `PATCH net` | | `net/ipv4/`, `net/ipv6/`, `net/core/` | Networking core | `netdev/net.git` | `PATCH net` | | `net/sctp/`, `net/dccp/`, `net/tipc/` | Networking | `netdev/net.git` | `PATCH net` | | `net/can/` | CAN | `linux-can/linux.git` | `PATCH can` | | `net/nfc/` | NFC | `sameo/nfc.git` | `PATCH nfc` | | `kernel/bpf/`, `net/bpf/` | BPF | `bpf/bpf.git` | `PATCH bpf` | | `drivers/net/ethernet/` | Network drivers | `netdev/net.git` | `PATCH net` | | `drivers/bluetooth/` | Bluetooth drivers | `bluetooth/bluetooth.git` | `PATCH bluetooth` | | `drivers/gpu/drm/` | DRM/GPU | `drm/drm.git` | `PATCH drm` | | `drivers/usb/` | USB | `usb/usb.git` | `PATCH usb` | | `sound/` | Sound/ALSA | `tiwai/sound.git` | `PATCH sound` | | `fs/ext4/` | ext4 | `tytso/ext4.git` | `PATCH ext4` | | `fs/btrfs/` | Btrfs | `kdave/btrfs.git` | `PATCH btrfs` | | `fs/xfs/` | XFS | `djwong/xfs-linux.git` | `PATCH xfs` | | `mm/` | Memory management | `akpm/mm.git` | `PATCH mm` | | `io_uring/` | io_uring | `axboe/linux-block.git` | `PATCH io_uring` | | `security/apparmor/` | AppArmor | `jj/linux-apparmor.git` | `PATCH apparmor` | | Others | General | `torvalds/linux.git` | `PATCH` | **The trap**: `net/bluetooth/` is under `net/` but does NOT go to `netdev/net.git`. Bluetooth patches go to `bluetooth/bluetooth.git` and are picked by the Bluetooth maintainer (Luiz Augusto von Dentz). Eventually they flow through `netdev/net.git` into mainline, but patches must be submitted to the Bluetooth tree directly. ``` WRONG: net/bluetooth/l2cap_core.c → "this is net/ → netdev/net.git → PATCH net" RIGHT: net/bluetooth/l2cap_core.c → get_maintainer.pl → bluetooth.git → PATCH bluetooth ``` **Another trap**: Many subsystems also maintain a `-next` tree for development patches (e.g., `netfilter/nf-next.git`, `netdev/net-next.git`, `bpf/bpf-next.git`). A fix may exist in **either** the fixes tree or the -next tree but not yet in `torvalds/linux.git`. **You MUST fetch and search both subsystem trees during deduplication — searching only `origin` (torvalds) will miss fixes that are queued in the subsystem but not yet merged into mainline.** ``` WRONG: git log origin/master --grep='nf_tables' -- net/netfilter/ ← only searches torvalds RIGHT: git fetch nf; git fetch nf-next git log nf/main nf-next/main origin/master --grep='nf_tables' -- net/netfilter/ ``` **Step 3: When in doubt, always trust `get_maintainer.pl`** ```bash # It handles all the edge cases in MAINTAINERS ./scripts/get_maintainer.pl --scm net/bluetooth/l2cap_core.c # Output will show bluetooth.git, not net.git ``` **Step 4: Identify the relevant kernel version** - Check if the bug exists in mainline, stable, or LTS - Parse from crash log: `grep 'Not tainted' crash.log` ### 1.4 Create the Analysis Plan Use Plan mode to structure the work. A typical plan: ``` 1. Clone source tree and checkout relevant version 2. [Parallel] Spawn subagents for: a. Static analysis: read the vulnerable code path, trace data flow b. Git archaeology: find the commit that introduced the bug (git log, git bisect) c. Cross-reference: search kernelctf knowledge base for similar vulnerability patterns 3. Set up QEMU environment with matching kernel config 4. Reproduce the crash with PoC 5. Dynamic analysis with GDB to confirm root cause 6. Assess exploitability 7. Develop and test patch 8. Package report and artifacts ``` ### 1.5 Subagent Dispatch Strategy This skill makes heavy use of subagents (the Agent tool) to parallelize work. The guiding principle: **plan centrally, execute in parallel, synthesize results**. **Parallel-safe tasks** (can run as concurrent subagents): - Source code reading of different files/functions - Git log / git blame on different paths - Searching the kernelctf knowledge base - Compiling kernel in a worktree - Web research for related CVEs or patches **Sequential tasks** (must wait for prior results): - Dynamic analysis depends on QEMU environment being ready - Patch writing depends on confirmed root cause - Patch verification depends on patch being applied When spawning subagents, always provide: - Clear, self-contained task description - All file paths and context needed (the subagent has no memory of your conversation) - Expected output format - Use `isolation: "worktree"` for tasks that modify files (compilation, patching) --- ## Phase 2: Source Acquisition & Static Analysis ### 2.1 Acquire and Update Source In practice, the user usually already has a local kernel source tree. Do NOT blindly clone a fresh repo every time — check what's available first. **Case A: User already has a local kernel source tree (most common)** ```bash cd /path/to/existing/kernel-src # 1. Check current state git status # any uncommitted changes? git describe --tags --abbrev=0 # what version is this? git remote -v # what remote does it track? # 2. Fetch latest from upstream — ALWAYS do this git fetch origin git fetch --tags origin # 3. Check how far behind we are git log --oneline HEAD..origin/master | head -20 # If significantly behind (>100 commits), strongly recommend updating # 4. Update to latest (if tree is clean) git pull --rebase origin master # Or if on a specific branch: git pull --rebase origin ``` If the user has uncommitted changes (their own annotations, previous patches, etc.): - `git stash` first, then fetch/pull, then `git stash pop` after analysis - Or work on a detached HEAD at the latest tag: `git checkout ` **Case B: No local source — clone fresh** ```bash # For the appropriate subsystem tree: git clone /path/to/analysis/kernel-src cd /path/to/analysis/kernel-src git checkout # For full history (needed for git bisect / git blame): git clone /path/to/analysis/kernel-src # Shallow clone is faster but limits git bisect: git clone --depth=1 --branch /path/to/analysis/kernel-src ``` **Case C: Local source exists but tracks a different tree** Sometimes the user has `torvalds/linux.git` but the bug is in a subsystem tree (e.g., `netdev/net.git`). Add it as a second remote: ```bash git remote add net git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git git fetch net git log net/master --oneline | head -5 ``` ### 2.2 Verify Source Version (Required — Do This Before Any Analysis) Writing a patch against stale source is a critical mistake: the patch may not apply to mainline, may fix an already-fixed bug, or may be semantically wrong due to context changes. Always verify the source is current before proceeding. **Step 1: Check if the local tree is up-to-date with remote AND fetch subsystem trees** ```bash git fetch origin git log HEAD..origin/master --oneline | head -10 # If output is non-empty → the local tree is behind. Update it: git pull --rebase origin master # CRITICAL: Also fetch the subsystem tree for the affected file. # Use the subsystem tree mapping table above, or run: # ./scripts/get_maintainer.pl --scm # to find the correct git tree URL. # # Example for net/netfilter/: # git remote add nf https://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf.git 2>/dev/null # git remote add nf-next https://git.kernel.org/pub/scm/linux/kernel/git/netfilter/nf-next.git 2>/dev/null # git fetch nf # git fetch nf-next # # Example for net/core/, net/ipv4/, drivers/net/: # git remote add net https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git 2>/dev/null # git remote add net-next https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git 2>/dev/null # git fetch net # git fetch net-next # # For other subsystems, follow the same pattern with the tree from the table. # The 2>/dev/null on `git remote add` suppresses "already exists" errors on re-runs. ``` **Step 2: Check if the bug is already fixed — search BOTH origin AND subsystem trees** This is the most important check — someone may have already submitted a fix. **A fix may exist in the subsystem tree (e.g., nf.git, net-next.git) but not yet in torvalds/linux.git.** Searching only origin will miss these. You MUST search all fetched remotes. ```bash # Determine which refs to search (adjust remote names per subsystem) # Example for netfilter — replace nf/main nf-next/main with your subsystem refs SEARCH_REFS="origin/master nf/main nf-next/main" # Search for patches touching the vulnerable function git log $SEARCH_REFS --oneline -S '' -- # Search for Fixes: tags referencing the introducing commit git log $SEARCH_REFS --oneline --grep='Fixes: ' # Search commit messages for the bug description keywords git log $SEARCH_REFS --oneline --grep='' -- # Check the linux-stable tree for backported fixes git log --all --oneline --grep='' ``` If a fix already exists (in ANY of the searched refs — origin OR subsystem tree): - **Report it** — tell the user the bug is already fixed, with the fixing commit hash and which tree it's in - **Verify the fix** — read the upstream fix to confirm it actually addresses the root cause - **Skip to Phase 7** — no need to write a new patch; document the existing fix in the report **Step 3: Two-Stage Workflow — Analyze on Crash Version, Patch on Latest** This is a **critical distinction** that the skill MUST enforce: Extract the crash kernel version from the log first: ```bash # Parse the crash log for the kernel version CRASH_VERSION=$(grep -oP 'Not tainted \K\S+' crash.log) # e.g., CRASH_VERSION="6.12.77" # Find the closest git tag CRASH_TAG=$(git tag -l "v${CRASH_VERSION}*" | sort -V | tail -1) # Or for stable kernels: git tag -l "v$(echo $CRASH_VERSION | cut -d. -f1-2)*" | sort -V | tail -1 # Get current latest upstream git fetch origin --tags LATEST=$(git describe --tags --abbrev=0 origin/master) ``` ``` ┌────────────────────────────────────────────────────────────────────┐ │ Crash log says kernel $CRASH_VERSION (e.g., a stable/old release) │ │ Latest upstream is $LATEST (e.g., mainline HEAD) │ │ │ │ WRONG: Write the patch against $CRASH_VERSION and call it done. │ │ RIGHT: Analyze on $CRASH_VERSION, then rebase the fix onto │ │ $LATEST mainline/subsystem-tree HEAD before finalizing. │ └────────────────────────────────────────────────────────────────────┘ ``` **Stage 1 — Analyze & Reproduce on the crash version**: ```bash # Checkout the crash kernel version for analysis and QEMU reproduction git checkout "$CRASH_TAG" # Build, boot in QEMU, reproduce the crash, do root cause analysis # This ensures you understand the bug in the exact context it was reported ``` **Stage 2 — Fetch latest code and check before writing the patch**: ```bash # ALWAYS fetch BOTH origin AND the subsystem tree before writing ANY patch git fetch origin git fetch --tags origin # Fetch the subsystem remote (must have been added in Phase 2.2 Step 1) # Example for netfilter: git fetch nf git fetch nf-next # Check: does the vulnerable code still exist in the latest version? git show origin/master: | grep '' # If the code has been refactored or removed → the bug may be moot in mainline # Check: has someone already fixed this in the latest tree? # CRITICAL: Search BOTH origin AND subsystem refs — a fix queued in the # subsystem tree may not have reached torvalds/linux.git yet. SEARCH_REFS="origin/master nf/main nf-next/main" # adjust per subsystem git log $SEARCH_REFS --oneline -S '' -- | head -10 git log $SEARCH_REFS --oneline --grep='' -- | head -10 # If bug still exists in latest → write patch against latest subsystem HEAD: git checkout origin/master # Or for networking fixes: git checkout net/main # netdev/net.git main branch # Or for netfilter fixes: git checkout nf/main # netfilter/nf.git main branch ``` **Stage 3 — Write the patch against the latest code**: ```bash # The patch MUST be based on the latest subsystem tree HEAD # NOT on the old crash kernel version git diff > patch.diff # your fix, based on latest code # Verify the fix also applies to the crash version (for QEMU testing) git stash git checkout "$CRASH_TAG" git stash pop # if it applies cleanly # Or: git cherry-pick / manual port if context differs ``` **Why this matters**: - Upstream WILL NOT accept patches based on old stable kernels - The code around the bug may have changed (variable renames, refactors, new callers) - A patch against an old stable may not apply to the latest mainline at all - Even if the patch applies, context lines may differ → `git am` fails | Crash kernel version | Analyze on | Write patch against | |---|---|---| | Stable (e.g., X.Y.Z) | `v$CRASH_VERSION` tag | Latest `origin/master` or subsystem HEAD | | Mainline (e.g., X.Y-rcN) | `v$CRASH_VERSION` tag | Latest `origin/master` | | Distro (e.g., X.Y.Z-distro) | Distro source | Upstream mainline HEAD | | net-next / subsystem tree | Subsystem HEAD | Subsystem HEAD (already latest) | **If the local tree is old and behind upstream**: ```bash # You MUST update before writing the patch git fetch origin git log --oneline HEAD..origin/master | wc -l # If significantly behind → pull or checkout latest # Check if the vulnerable file has changed significantly git diff "$CRASH_TAG"..origin/master -- | diffstat # If large diff → the patch context has changed, must write against latest ``` **Step 4: Record the base commit in the report** Always document the exact commit your patch is based on: ```bash echo "Patch base: $(git log --oneline -1 HEAD)" >> report_metadata.txt # e.g., "Patch base: abc123def456 Merge tag 'net-6.12-rc4'" ``` This goes in the report's metadata section so anyone applying the patch knows exactly which tree state it was developed against. **If the source tree was already present** (e.g., the user has a local clone): - Still run Steps 1-3 — don't assume it's current - `git fetch && git log HEAD..origin/master --oneline` is fast and catches stale trees - If the tree is weeks/months behind, warn the user before proceeding --- ## GATE CHECK: Before Entering Phase 3 — Source Version Verified **DO NOT proceed to Phase 3 (Root Cause Analysis) until the source tree version is verified.** This gate ensures you are analyzing and will patch against the correct code. ### Required Checks (must ALL pass) ```bash echo "=== Source Version Gate Check ===" # 1. Was remote fetched? echo -n "1. Remote fetched: " git fetch origin --tags 2>&1 | tail -1 echo "DONE" # 1b. Was SUBSYSTEM remote fetched? # Identify the subsystem remote from Phase 2.2 Step 1 (e.g., nf, nf-next, net, bpf) echo -n "1b. Subsystem remote fetched: " # Replace SUBSYS_REMOTE / SUBSYS_NEXT with the actual remote names SUBSYS_REMOTE="nf" # adjust per subsystem SUBSYS_NEXT="nf-next" # adjust per subsystem (may not exist for all) git fetch "$SUBSYS_REMOTE" 2>&1 | tail -1 && echo " $SUBSYS_REMOTE DONE" git fetch "$SUBSYS_NEXT" 2>&1 | tail -1 && echo " $SUBSYS_NEXT DONE" # If `git remote` does not list the subsystem → go back to Phase 2.2 Step 1 # 2. What is the latest upstream tag? echo -n "2. Latest upstream tag: " LATEST=$(git describe --tags --abbrev=0 origin/master 2>/dev/null) echo "$LATEST" # 3. Are we on or ahead of the latest tag? echo -n "3. Source version: " BEHIND=$(git log --oneline HEAD..origin/master 2>/dev/null | wc -l) if [ "$BEHIND" -eq 0 ]; then echo "PASS — up to date ($(git describe --tags HEAD))" else echo "FAIL — $BEHIND commits behind $LATEST" echo " → Run: git checkout $LATEST" fi # 4. Has the vulnerable file changed between current HEAD and latest? echo -n "4. Vulnerable file changed since HEAD? " VFILE="" CHANGES=$(git diff --stat HEAD..origin/master -- "$VFILE" 2>/dev/null | grep -c '|') if [ "$CHANGES" -eq 0 ]; then echo "PASS — no changes" else echo "WARNING — file changed, MUST patch against latest" fi # 5. Bug already fixed — search origin AND subsystem trees? echo -n "5. Already fixed? " SEARCH_REFS="origin/master" # Add subsystem refs if they exist git rev-parse --verify "$SUBSYS_REMOTE/main" &>/dev/null && SEARCH_REFS="$SEARCH_REFS $SUBSYS_REMOTE/main" git rev-parse --verify "$SUBSYS_NEXT/main" &>/dev/null && SEARCH_REFS="$SEARCH_REFS $SUBSYS_NEXT/main" echo " Searching: $SEARCH_REFS" git log $SEARCH_REFS --oneline -S '' -- "$VFILE" | head -5 ``` **If check 1b shows subsystem remote not configured → go back to Phase 2.2 Step 1.** **If check 3 shows FAIL → checkout the latest tag before proceeding.** **If check 5 shows a fix already exists (in ANY ref — origin OR subsystem tree) → report the existing fix, skip to Phase 7.** ### Common Mistakes This Gate Prevents | Mistake | Consequence | This gate catches it | |---|---|---| | Patching against stale tree | Patch may not apply to mainline | Check 3 | | Missing a context change | Patch applies but is semantically wrong | Check 4 | | Duplicate work (mainline) | Bug already fixed in torvalds/linux.git | Check 5 (origin) | | Duplicate work (subsystem) | Fix queued in subsystem tree, not yet in mainline | Check 5 (subsystem refs) | | Subsystem remote not fetched | Miss fixes in nf.git / net-next.git / etc. | Check 1b | | Wrong version in report | Report claims wrong "latest affected" | Check 2 | --- ### 2.3 Static Analysis (Spawn as Subagents) Launch these in parallel: **Subagent A — Code Path Analysis**: - Read the functions in the call stack, starting from the crash point - Trace the data flow: where does the faulting pointer come from? - Identify the object lifecycle: allocation, use, free - Look for missing locks, reference count issues, error path leaks **Subagent B — Git Archaeology**: - `git log --oneline ` for recent changes to the affected files - `git blame` on the vulnerable lines to find the introducing commit - Check if there are already patches in mainline or -next that fix this - Look for related fixes in the same area (often bugs cluster) **Subagent C — Knowledge Base Cross-Reference**: - Search `references/kernelctf-knowledge-base.md` for similar vulnerability patterns - Search `references/vuln-classification.md` to classify the bug type - Check if this subsystem has known exploit primitives --- ## Phase 3: Root Cause Analysis This is the most critical phase. The symptom (what the crash log shows) often differs from the actual bug. **Common Symptom-vs-Root-Cause Mismatches**: | Crash Symptom | Possible True Root Cause | |---|---| | NULL pointer dereference | UAF (object freed, memory reused/zeroed) | | General Protection Fault | UAF (object freed, slab poisoned with 0x6b6b6b6b) | | KASAN: slab-use-after-free | Straightforward UAF, but find the race condition | | KASAN: slab-out-of-bounds | Off-by-one, integer overflow leading to undersized allocation | | BUG: unable to handle page fault | UAF, double-free, or type confusion | | WARNING in refcount_t | Reference count underflow — likely a UAF waiting to happen | | UBSAN: shift-out-of-bounds | Integer handling bug, possibly exploitable for info leak | **Always ask**: "What is the actual invariant violation, not just the symptom?" Read `references/vuln-classification.md` for the full taxonomy of kernel vulnerability classes and how to distinguish them. ### 3.1 Determine the True Bug Class To identify the real root cause: 1. **Trace object lifetime**: When was the object allocated? When freed? Who still holds a reference? 2. **Identify the race window**: For concurrency bugs, what's the race between? (syscall vs IRQ, two CPUs, etc.) 3. **Check error paths**: Many kernel bugs live in error handling — a `goto err` that forgets to unlock or drop a reference 4. **Verify with KASAN alloc/free stacks**: If KASAN provides them, the allocation and free call stacks tell you exactly who created and destroyed the object ### 3.2 Build the Bug Narrative — Source-Level Deep Trace (Required) A shallow narrative ("Thread A frees, Thread B uses" or "missing NULL check") is NOT sufficient. After confirming the root cause, you must produce a **source-level deep trace** that combines the kernel source code and the PoC's behavior to explain exactly how the bug manifests. This trace must be generated **from the actual source code and PoC for each specific bug**. The approach varies by vulnerability class — use the matching methodology below. #### Common Steps (All Vulnerability Types) **Step A: Map PoC actions to kernel code paths** Read the PoC and trace what each part does in the kernel: - `PoC line/action → syscall/packet → kernel entry function (file.c:line)` - For multi-threaded PoCs: which thread does what, and what's the intended race **Step B: Read the relevant kernel source with file:line citations** For every function in the call chain from syscall entry to crash: - What does it do? What data does it read/write? - What validation/checks does it perform (or fail to perform)? - What synchronization (locks, RCU, refcount, memory barriers) does it use? **Step B.1: Verify reachability — trace call-chain preconditions** For each code site identified as vulnerable, verify that the bad state can actually occur there. Trace the full call chain backwards and identify all preconditions that must hold for execution to reach that site: feature flags, device configuration constraints, creation-time validation, compile-time guards, caller-enforced invariants. If a precondition already guarantees the state is valid at that site, the site is not truly vulnerable — exclude it from the fix. Do NOT assume symmetry (e.g., "if the IPv6 path needs a fix, the IPv4 path must too") without proving each case independently. **Step C: Build a timeline/flow diagram from source** Produce a visual that shows the bug's progression. The format depends on the vulnerability class (see below). Use actual function names and file:line references from the source, not generic placeholders. **Step D: Explain why the bug exists** - What invariant is violated? - What mechanism was supposed to prevent this? Why did it fail? - Is this a design flaw or an implementation oversight? #### Per-Vulnerability-Class Methodology **Choose the methodology that matches your bug's root cause.** Not every bug involves refcounts or races — trace what's actually relevant. **UAF / Double-Free / Refcount bugs**: - Trace the object's full refcount lifecycle: creation (init → get) → normal state → destruction (put → release → free), with refcount value at each step - Identify who holds each reference and when they release it - Show the race timeline (side-by-side CPUs) with refcount transitions as `before→after` - Highlight: missing `kref_get_unless_zero`, premature `put`, or unprotected reader **Race conditions (non-refcount)**: - Identify the shared state being raced on (flag, pointer, list, counter) - Show the TOCTOU window: what's checked, what changes, what's used - Side-by-side CPU timeline showing interleaving that leads to the bug - Highlight: missing lock, wrong lock scope, missing memory barrier **NULL pointer dereference (non-race)**: - Trace the data flow: where does the NULL pointer originate? - Is it from a failed allocation? A missing initialization? An error path that skips setup? A sparse array lookup (like `inet_protos[]`)? - Show the call chain from the point where NULL enters to the crash dereference - Highlight: what validation is missing and where it should be **Out-of-bounds (OOB) read/write**: - Trace the buffer allocation: what size, from where, based on what input? - Trace the access: what index/offset, from where, based on what input? - Show the arithmetic: `allocated_size` vs `accessed_offset` — why does it overflow? - If integer overflow: show the multiplication/addition that wraps - Highlight: missing bounds check, wrong size calculation, signedness confusion **Logic bugs / State machine errors**: - Map out the state machine: what states exist, what transitions are valid? - Show the sequence of operations that reaches an "impossible" state - Trace the error path that skips a required state transition - Highlight: missing state check, wrong transition order, error path that forgets cleanup **Info leaks**: - Trace the data flow from kernel memory to user space - Identify the uninitialized field, padding bytes, or stale pointer - Show the struct layout with `pahole` — which bytes are leaked? - Highlight: missing memset/initialization, struct padding, wrong copy size **Type confusion**: - Show the two types involved and their different layouts - Trace how the object gets cast/reinterpreted as the wrong type - Highlight which fields overlap incorrectly (especially function pointers vs data) #### Step E: Visualize Your Analysis with ASCII Diagrams Diagrams are NOT a separate step — they are the **visual output of the source analysis above**. After completing Steps A-D, produce diagrams that summarize what you found. The diagrams must reference actual function names and file:line from YOUR analysis, not generic templates. Generate whichever diagram types are relevant to the bug: - **Call chain + data transformation**: Show how data flows through each function with `skb->data` / pointer / buffer state at each layer. Each box = actual function (file:line). - **Race timeline (for concurrency bugs)**: Side-by-side CPUs with actual function names, refcount/state transitions, and the race window marked. - **Struct layout (for OOB/type confusion/info leak)**: pahole-style field offsets showing which field is corrupted/leaked/confused. Use actual struct name from the source. - **Packet/data format (for protocol bugs)**: Byte-level layout of attacker input showing which fields are controlled and where validation is missing. - **Object lifecycle (for UAF)**: Allocation → use → free → use-after-free with refcount values at each step, referencing actual functions. - **State machine (for logic bugs)**: Valid vs actual state transitions. - **Memory/slab layout (for heap bugs)**: Slab page showing adjacent objects. **The diagram must reflect your source code analysis — not be a generic template.** For example, a call chain diagram should use the real function names you found in Step B, not placeholder names like `function_a()`. #### Quality Criteria (All Types) - Every source reference has a **file:line** citation - The PoC's behavior is mapped to kernel code paths - State changes (refcount, lock, flag, pointer) show **before→after** values - The crash is traced to a specific struct field and offset - Diagrams use actual function/struct names from the source analysis, not placeholders - There's a clear explanation of **what's broken and why** #### What to Avoid - Generic descriptions without source references ("the object is freed then used") - Drawing diagrams without doing the source analysis first (diagrams are OUTPUT, not INPUT) - Using only one methodology for all bug types (not everything is a refcount race) - Skipping the PoC→kernel mapping (the reader needs to understand HOW the bug triggers) - Copying template diagrams instead of generating them from your analysis ### 3.3 Determine Affected Version Range After identifying the introducing commit, determine the exact affected version range: ```bash # Find the earliest release tag containing the introducing commit git tag --contains | sort -V | head -5 # e.g., v3.13-rc1 → bug exists since v3.13 # If a fix already exists upstream, find when it landed git tag --contains | sort -V | head -5 # e.g., v6.14-rc2 → fixed in v6.14 # Check which stable branches are affected git branch -r --contains | grep 'stable' # Check which stable branches have the fix backported git branch -r --contains | grep 'stable' ``` Record in the report: `Affected: v3.13 — v6.13 (fixed in v6.14-rc2)` For stable/LTS impact, check if the fix needs `Cc: stable@vger.kernel.org`. ### 3.4 Diagram Reference (Format Examples) When producing diagrams in Step E above, use these format conventions. These are **formatting templates only** — your actual diagrams must use real function names and data from your source analysis. See `references/crash-log-analysis.md` for address interpretation patterns and `references/vuln-classification.md` for the bug classification decision tree. ### 3.5 Validation-Layer Bugs: Comprehensive Verification Analysis **When this applies**: the buggy code's job is to parse, decode, or deserialize *untrusted or on-disk data* into in-memory structures that later code trusts — log/journal recovery, on-disk metadata readers, netlink/attribute parsers, packet/protocol decoders, ELF/module loaders, ioctl argument unpacking, restore-from-checkpoint paths. For these bugs the single missing check the crash revealed is almost never the whole bug. The code has a **contract** — "produce only well-formed structures from arbitrary input" — and a crash means that contract is under-enforced in general. **Do NOT spot-fix.** Adding the one check the crash exposed, and shuffling the existing checks into a helper, is the most common way to get a validation-layer patch rejected by a maintainer — it looks like refactoring, not hardening, and signals shallow analysis. The correct response is to analyze the **entire layer** and design verification that captures every way the input can be malformed, in one coherent abstraction. **The framing to apply to yourself (and to any Explore/analysis subagent you dispatch)** — point it at the whole file/function, not the crash line: > Assume every byte this code reads from {disk/journal/socket/user} is > attacker-controlled and may be arbitrarily malformed. The structures built here > are later trusted and dereferenced. Enumerate *every* issue the current > validation fails to catch — not just the one that crashed — and propose a > layered plan to robustly verify the data before it is used. This is the prompt that turns the model from a spot-fixer into an auditor. #### Step 1: Map the trust boundary and the bootstrap fields - Where does untrusted data enter, and at what point is the assembled structure first *used/trusted* (dereferenced, used to size an allocation, used as an index)? That transition is the **trust boundary** — verification belongs there, before first use, not scattered downstream. - Identify the **bootstrap / self-describing fields**: the fields read *first* so the code knows how to parse the rest — a type tag, a region/element count, a length, an offset. These are the highest-value targets: everything downstream trusts them to build structures. A malformed count is how "read a count from disk → allocate/loop on it" becomes an OOB or overflow. #### Step 2: Find the "commit point" — verify when you stop trusting more input Incremental parsers accumulate input piece by piece (region by region, attribute by attribute, TLV by TLV). There is a specific point where the code *decides it will consume no more input for the current item* and finalizes it. That commit point is where whole-object verification belongs: the header, layout, counts, and total assembled size are all known there, and nothing downstream has consumed them yet. Verify the **finishing** object before starting the next one, and comprehensively verify the **new** object's header before trusting it to drive further parsing. #### Step 3: Enumerate every missing check (be exhaustive, whole-layer) For the entire layer, list what is currently trusted-without-checking. Typical categories: - **Known-type check** — is the type tag a known value? (Reject unknown *before* using it to dispatch or size anything.) - **Count bounds** — is the element/region count within the maximum valid for *this* type? (Not merely "nonzero" — the per-type maximum.) - **Per-element size** — given the now-known type, is each element's size within the max any element of that type may have? - **Total-size / layout consistency** — does the sum of parts match the declared whole? Do offsets stay in bounds? Is required alignment satisfied? - **Structural minimum** — is each buffer at least `sizeof(the struct we cast it to)` before the cast/field access? - **Premature-terminator / out-of-sync detection** — do we hit a new header or terminator before the current object was expected to end? - **Underflow / NULL on continuation** — for incremental assembly, is the accumulator non-empty and allocated before we append to `[count - 1]`? Produce this as a table (site → what's trusted → how it can be malformed → what check is missing), covering the whole file, not just the crash path. #### Step 4: Design the layered abstraction (before any code) Collapse the enumerated checks into a **clean two-tier design** — mixing the tiers is what makes verification code incomplete and unmaintainable: - **Generic / structural verification** (type-agnostic): known-type, count vs per-type max, per-region size sanity, alignment, minimum sizes, out-of-sync detection. Runs once at the commit point for every item. - **Type-specific / semantic verification**: each item/message type validates its own region count and per-region sizes against what *that* type requires, before its consumer casts and dereferences. Often best as a per-type `verify()` callback in the existing ops table. State explicitly which checks live in which tier and where each is called from. **This design — not the individual checks — is the Phase 3 deliverable for a validation-layer bug.** Confirm it at the high level (Phase 6.1 "Design before code") before writing a line. #### What "good" looks like vs the rejected spot-fix | Rejected (spot-fix) | Accepted (comprehensive) | |---|---| | Add the one missing check the crash exposed | Enumerate every way the input can be malformed across the whole layer | | Move existing scattered checks into a helper, unchanged | Design two tiers (generic structural + per-type) placed at the trust/commit boundary | | Fix only the crash path | Cover every path that builds structures from this input | | "It no longer crashes on the PoC" | "Malformed input of any of these N forms is rejected before use" | Read `references/verification-hardening.md` for the full methodology and a worked example. --- ## Phase 4: Dynamic Analysis (QEMU + GDB) ### 4.1 Set Up QEMU Environment Read `references/qemu-setup.md` for detailed setup instructions. Key requirements: - Build kernel with: `CONFIG_KASAN=y`, `CONFIG_DEBUG_INFO=y`, `CONFIG_GDB_SCRIPTS=y`, `CONFIG_FRAME_POINTER=y`, `CONFIG_HARDENED_USERCOPY=y` (and relevant subsystem configs) - Use `virtme-ng` for quick boot if applicable, or full QEMU with custom rootfs - Prepare a minimal rootfs with the PoC compiled and ready ```bash # Example QEMU launch (adjust as needed) qemu-system-x86_64 \ -kernel arch/x86/boot/bzImage \ -initrd rootfs.cpio.gz \ -append "console=ttyS0 root=/dev/ram rdinit=/init nokaslr" \ -nographic \ -m 2G \ -smp 2 \ -s -S # GDB stub on port 1234, halt at start ``` ### 4.2 Reproduce and Debug 1. Boot the vulnerable kernel in QEMU 2. Run the PoC and confirm the crash reproduces 3. Attach GDB: `gdb vmlinux -ex "target remote :1234"` 4. Set breakpoints at key functions identified in static analysis 5. Step through the vulnerable code path 6. Confirm the root cause hypothesis from Phase 3 ### 4.3 Key GDB Commands for Kernel Debugging ``` # Kernel-specific lx-symbols # Load kernel module symbols lx-dmesg # Print kernel log lx-ps # List processes lx-lsmod # List modules # Analysis p *(struct sk_buff *)$rdi # Print kernel structures info threads # Check CPU/thread state bt # Backtrace watch *(int *)0xaddr # Hardware watchpoint on the vulnerable field ``` ### 4.4 Handling Non-Deterministic Reproduction Race conditions and timing-sensitive bugs may not crash on every run. **Increase reproduction rate**: ```bash # Loop the PoC — run 100 times and count crashes for i in $(seq 1 100); do timeout 5 ./poc 2>/dev/null echo "Run $i: exit=$?" done # Increase CPU count to widen race windows qemu-system-x86_64 ... -smp 4 # or -smp 8 # Add system stress to increase scheduling pressure stress-ng --cpu 4 --io 2 --vm 2 --timeout 60 & ./poc # Use taskset to pin PoC threads to specific CPUs taskset -c 0,1 ./poc ``` **Record reproduction rate** in the report: e.g., "Triggers 30/100 runs with -smp 4" **If the PoC never crashes**: - Verify kernel config matches the crash environment (especially KASAN, PREEMPT, SMP) - Check if the compiler version matters (see `references/syzbot-workflow.md`) - Try the exact syzbot kernel config if available - Add `usleep()` delays in the PoC to manipulate race timing - Use `ftrace` to confirm the race window exists even if it doesn't crash --- ## Phase 5: Exploitability Assessment After confirming the root cause, assess whether this bug is exploitable for privilege escalation, information leak, or denial of service. Read `references/exploitability-assessment.md` for the full assessment framework. ### 5.1 Key Questions 1. **What primitive does this bug give an attacker?** - UAF → potential arbitrary read/write via heap spray - OOB write → adjacent object corruption - Double-free → overlapping allocations - Info leak → KASLR bypass - Race condition → how wide is the window? Is it winnable? 2. **What is the attack surface?** - Reachable from unprivileged user? Needs `CAP_NET_ADMIN`? Needs user namespaces? - Reachable from network? From a container? 3. **What objects share the same slab cache?** - For UAF/OOB: what useful kernel objects (e.g., `struct cred`, `struct file`, `msg_msg`, `pipe_buffer`, `sk_buff`) live in the same `kmalloc-*` bucket? - Can the attacker control allocation/free timing? 4. **What mitigations apply?** - KASLR, SMEP, SMAP, CFI, RANDSTRUCT - `CONFIG_SLAB_FREELIST_RANDOM`, `CONFIG_SLAB_FREELIST_HARDENED` - `CONFIG_HARDENED_USERCOPY`, `CONFIG_USERFAULTFD` availability 5. **Is there precedent?** - Search the kernelctf knowledge base for exploits in the same subsystem or using similar primitives - Reference known techniques: `msg_msg` spray, `pipe_buffer` ROP, `io_uring` primitives, cross-cache attacks ### 5.2 Permission Gate Analysis (`capable` vs `ns_capable`) — Required Before rating exploitability, you MUST trace the full permission check chain from syscall entry to the vulnerable function. This determines the true attack surface. **Why this matters**: A bug gated by `capable(CAP_NET_ADMIN)` needs real root. But the SAME capability checked via `ns_capable()` is obtainable by any unprivileged user through `unshare(CLONE_NEWUSER|CLONE_NEWNET)`. Many analysts miss this distinction, leading to wrong severity assessments. **Step 1: Find all capability checks in the call path** ```bash # Trace from syscall entry to the vulnerable function # and grep for every permission check along the way grep -n 'capable\|ns_capable\|netlink_capable\|nfnl.*capable\|sk_net_capable' \ net/netfilter/nfnetlink.c net/netfilter/nfnetlink_osf.c ``` **Step 2: Draw the permission gate diagram** For each layer, document: what check, namespace-aware or not, what happens on failure. ``` User-space syscall (sendmsg on netlink socket) │ ▼ Framework layer (e.g., nfnetlink_rcv) │ netlink_net_capable(skb, CAP_NET_ADMIN) ← namespace-aware (ns_capable) │ userns root CAN pass this gate │ failure → -EPERM, never reaches callback │ ▼ Subsystem callback (e.g., nfnl_osf_add_callback) │ capable(CAP_NET_ADMIN) ← init_user_ns ONLY │ userns root CANNOT pass this gate │ failure → -EPERM │ ▼ Vulnerable code path ``` **Step 3: Determine effective privilege requirement** The effective requirement is the **most restrictive** check in the entire chain: - If ANY layer uses `capable()` → needs real root (init_user_ns) - If ALL layers use `ns_capable()` / `netlink_capable()` → reachable via userns - Watch for framework vs callback mismatch (common pattern: framework is ns-aware but specific callback adds a stricter `capable()` check) **Step 4: Document in the report** Include the ASCII gate diagram in the report's exploitability section. State clearly: - "Effective privilege: unprivileged (all checks are namespace-aware)" OR - "Effective privilege: real root (callback uses `capable()` at line X)" - Note if a future `capable()` → `ns_capable()` change would expand the attack surface Read `references/exploitability-assessment.md` § "Critical: capable() vs ns_capable()" for the full reference on which subsystems use which check. ### 5.3 Challenge Your Initial Assessment (Required) **Do NOT stop at the first conclusion.** The most common mistake in kernel vulnerability analysis is accepting the surface-level classification ("it's just a NULL deref / DoS") without probing deeper. Many high-severity CVEs were initially dismissed as DoS. After forming your initial assessment, systematically challenge it by asking these questions: #### "Is the crash symptom hiding a stronger primitive?" | Initial symptom | Ask yourself | Deeper reality? | |---|---|---| | NULL ptr deref | Is KASAN enabled? Without KASAN, a UAF to zeroed memory looks like NULL deref | Possibly UAF | | NULL ptr deref at offset N | Is N controllable by the attacker? If so, this may be an arbitrary-address read | Possible info leak | | GPF / non-canonical addr | Is the address derived from attacker data + corruption? Could the attacker make it canonical? | Possible controlled dereference | | Single-byte OOB write | What's adjacent in the slab? Even 1 byte can flip a boolean flag or corrupt a refcount | Possible privilege escalation | | Refcount WARNING | Does this eventually lead to a UAF if triggered enough times? | Likely UAF with patience | | DoS-only race | With different timing, does the race give a wider window? Can userfaultfd freeze the race? | Possible reliable UAF | #### "Is the NULL deref masking a UAF?" — The RCU Lifetime Pattern This is the most commonly missed upgrade path. When you see a NULL deref in RCU-protected code, ask: **where did the NULL come from?** ``` Pattern: RCU object has field cleared before grace period expires Teardown path (writer): Read path (RCU reader): ────────────────────── ──────────────────────── hlist_del_rcu(&obj->node) rcu_read_lock() obj = rcu_dereference(hash[idx]) obj->ops->destroy(obj) │ │ │ ← obj is valid (RCU protects it) ├─ resource_put(obj->ptr) │ but obj->ptr is being destroyed ├─ obj->ptr = NULL ◄── HERE │ │ │ └─ call_rcu(&obj->rcu, free_fn) obj->ptr->field ← NULL DEREF rcu_read_unlock() ``` **The critical question**: What happens in the window BETWEEN `resource_put(obj->ptr)` and `obj->ptr = NULL`? ``` Timeline: T1: resource_put(obj->ptr) → obj->ptr's refcount drops to 0 → memory freed T2: ───────────────────────── WINDOW: obj->ptr points to FREED MEMORY T3: obj->ptr = NULL → now it's "safely" NULL T4: call_rcu(free_fn) → obj itself deferred If reader accesses obj->ptr between T1 and T3 → UAF (not NULL deref!) If reader accesses obj->ptr after T3 → NULL deref (the "safe" crash) ``` **The PoC that crashes with NULL deref is hitting the T3→T4 window.** But the T1→T3 window is more dangerous — it's a real UAF where the reader follows a dangling pointer to freed memory. **How to evaluate this**: 1. Read the teardown code — is there a `put`/`release`/`free` BEFORE the `= NULL`? 2. If yes: the freed memory could be reallocated with attacker-controlled content 3. What object is being freed? What slab cache? What size? 4. Can the attacker spray that cache between T1 and T3? 5. What fields does the reader dereference? Function pointers? Data? **Hypothetical example — generic subsystem teardown**: ``` some_subsystem_destroy(obj): dev_put(obj->netdev, ...) ← T1: netdev refcount → 0, memory freed ──────────────────────────────── ← WINDOW: obj->netdev is dangling pointer obj->netdev = NULL ← T2: "safe" NULL call_rcu(&obj->rcu, obj_free) ← T3: obj itself deferred Reader hitting T1-T2 window: obj->netdev → freed memory → if reader dereferences function pointers through it → code execution → MUCH more dangerous than the NULL deref at T2+ ``` Look for this pattern whenever you see a NULL deref in RCU-protected code where the NULL comes from an explicit assignment in a teardown/destroy path. **If you find this pattern, the bug upgrades from "DoS" to "Likely/Highly Exploitable".** #### "Can I get a different/stronger primitive from the same root cause?" — Root-Cause-Driven Path Enumeration (Required) **This is the most critical and most often skipped step in exploitability assessment.** The PoC crashes on ONE code path. But the root cause — the underlying invariant violation — may be reachable through MULTIPLE code paths, each yielding a different primitive. You MUST enumerate all paths, not just analyze the crash path. **The methodology is: Root Cause → All Affected Sites → Per-Site Primitive → Pick Strongest.** ##### Step 1: Precisely define the root cause as a pattern Do NOT define the root cause as "crash in function X". Define it as the abstract invariant violation that CAUSES the crash. Examples: | Bad (crash-specific) | Good (root-cause pattern) | |---|---| | "NULL deref in `subsystem_periodic_scan`" | "Any RCU traversal of the object hash that dereferences `obj->parent` without first taking a reference via `kref_get_unless_zero()` can observe an object with refcount=0 and cleared parent pointer" | | "UAF in `subsystem_add_entry`" | "Any path that holds a stale pointer to an entry across a transaction commit/abort boundary can access freed memory" | | "OOB write in `subsystem_set_params`" | "Any path that uses user-provided parameters without validating `count * element_size` against the allocated buffer can write past the buffer" | The pattern definition determines your search scope. Be precise. ##### Step 2: Find ALL code sites matching the pattern Use grep/Explore agents to systematically enumerate every site: ```bash # Find all callers of the function that dereferences the unsafe pointer grep -rn '' / # Find all RCU traversals of the hash that DON'T use kref_get_unless_zero grep -rn 'hlist_for_each_entry_rcu.*' / ``` For each call site found, classify it: | Site | Context | Has refcount guard? | Vulnerable? | |---|---|---|---| | `periodic_scanner()` LNNN | workqueue (periodic) | No | **YES** | | `hash_find_helper()` LNNN | various callers | Yes (`kref_get_unless_zero`) | No | | `packet_rx_handler()` LNNN | packet RX path | Via `hash_find` | No | | ... | ... | ... | ... | **Do NOT stop at the first vulnerable site.** Exhaustively check every site. ##### Step 3: For each vulnerable site, analyze the exploitation primitive For every site marked "Vulnerable" in step 2, answer: 1. **What context does this code run in?** - Process context? → attacker can control scheduling, use userfaultfd - softirq/IRQ? → preemption disabled, harder to race but no context switch - Workqueue? → process context with possible sleep points 2. **What operations does this code perform on the vulnerable object?** - Read a field → info leak primitive? - Write a field → corruption primitive? - Dereference a pointer → if controllable, arbitrary read/write? - Call a function pointer → code execution? - Compare fields → bypass of security check? - Copy data to userspace → info leak to attacker? 3. **What objects/resources does this code touch AFTER the vulnerable access?** - Even if the initial access is a crash, is there a code path where the vulnerable access succeeds (non-NULL, valid memory) and continues to do something useful to the attacker? 4. **Can the attacker control what's in the freed/corrupted memory?** - What slab cache is the freed object in? - What's the object size? → which kmalloc bucket? - Can the attacker spray that cache via `msg_msg`, `sk_buff`, `setxattr`, etc.? - If so, what fake field values would give the attacker a useful primitive? 5. **What is the race window width?** - Back-to-back instructions → very narrow, but userfaultfd/FUSE can help - Separated by lock acquire → lock contention can widen the window - Separated by blocking operation → wide window, easy to exploit ##### Step 4: Cross-path comparison — find the strongest primitive Build a summary table: ```markdown | Vulnerable path | Primitive | Controllability | Race window | Verdict | |---|---|---|---|---| | `periodic_scanner` (workqueue) | NULL deref → DoS | None | periodic interval | DoS only | | `packet_handler` (if vulnerable) | Read parent fields → info leak? | Spray-dependent | Per-packet | Potential info leak | | `teardown_helper` (put-before-del) | UAF if obj freed while in hash | Spray target cache | Lock-width | Potential UAF | ``` The strongest primitive across ALL paths is the bug's true exploitability ceiling. ##### Step 5: Analyze pre-crash windows on the strongest path For the most promising path, trace the exact teardown sequence and identify ALL timing windows (not just the one the PoC hits): ``` Teardown sequence for the strongest vulnerable path: T0: [precondition] — attacker sets up the race T1: resource_put() — object freed, pointer dangling ← UAF WINDOW OPENS T2: ptr = NULL — pointer cleared ← NULL WINDOW (PoC crashes here) T3: call_rcu(free_fn) — parent object deferred free ← MEMORY FREED AFTER GRACE PERIOD Attacker wants to hit T1→T2 (UAF), not T2→T3 (NULL deref). ``` For each window, assess: - How wide is it? (instructions? microseconds? milliseconds?) - Can it be widened by the attacker? (CPU pinning, interrupt flooding, lock contention) - Can `userfaultfd` or `FUSE` freeze a page fault in the middle to hold the window open? - What specific slab object can be sprayed into the freed slot during this window? **The goal is not to confirm the PoC's crash — it's to find the strongest possible primitive from the root cause across all affected code paths.** #### "Can this be chained with other bugs?" Even a "DoS-only" bug can be valuable in a chain: - **Info leak + this bug**: If you have a separate KASLR bypass, does this bug become exploitable? - **This bug + another write**: A read primitive from this bug + a write primitive from another bug = full exploit - **This bug enables another**: Does crashing this specific code path leave the system in a state that makes another bug reachable? #### "What happens on different kernel configurations?" - Without `CONFIG_INIT_ON_FREE_DEFAULT_ON`: freed memory retains old data → a NULL deref might become a valid-pointer dereference (UAF) - Without KASAN: UAF doesn't get caught immediately → attacker has more time to spray - With `CONFIG_USERFAULTFD=y`: race windows can be frozen → narrow races become reliable - With older kernels: NULL page may be mappable (`mmap_min_addr=0`) → NULL deref becomes code execution #### Document your reasoning In the report, include a **full path enumeration section** like: ```markdown ### Root-Cause Path Enumeration Root cause pattern: {{precise description of the invariant violation}} #### All affected code sites | # | Function (Line) | Context | Refcount guard? | Vulnerable? | |---|---|---|---|---| | 1 | `func_a()` L123 | workqueue | No | **YES** | | 2 | `func_b()` L456 | process ctx | Yes (kref_get_unless_zero) | No | | 3 | `func_c()` L789 | softirq | No | **YES** | #### Per-path primitive analysis **Path 1: `func_a()` (workqueue context)** - Operations after vulnerable access: {{what the code does with the object}} - Primitive: {{DoS / info leak / UAF / arbitrary write / code exec}} - Race window: {{width, widenable?}} - Spray target: {{slab cache, object size, feasibility}} **Path 3: `func_c()` (softirq context)** - Operations after vulnerable access: {{...}} - Primitive: {{...}} - Race window: {{...}} - Spray target: {{...}} #### Strongest primitive across all paths {{The most dangerous path is #N because... / All paths are DoS-only because...}} ### Alternative Exploitation Analysis Initial assessment: {{initial primitive}} → {{initial impact}} Challenges considered: 1. Could this be a UAF? — {{Yes/No: specific reasoning citing lock/teardown analysis}} 2. Is the offset controllable? — {{Yes/No: trace where the offset comes from}} 3. Different timing (pre-NULL vs post-NULL window)? — {{Yes/No: analyze teardown sequence}} 4. Different trigger path? — {{Yes/No: reference path enumeration above}} 5. Chain potential? — {{Yes/No: what would be needed}} 6. Different kernel config? — {{Yes/No: INIT_ON_FREE, USERFAULTFD, mmap_min_addr}} Conclusion: {{Assessment stands / upgraded to X because path #N gives Y primitive}} ``` **If you find a stronger primitive**, update the rating and exploitation path accordingly. This step often upgrades "DoS" bugs to "Likely Exploitable" or higher. ### 5.4 Exploitability Rating Rate as one of: - **Highly Exploitable**: Reliable UAF/OOB-write with good heap spray target, reachable unprivileged - **Likely Exploitable**: Bug gives useful primitive but exploitation has challenges (narrow race, limited control) - **Potentially Exploitable**: Bug exists but exploitation path unclear or requires unusual conditions - **Unlikely Exploitable**: DoS only, confirmed no stronger primitive after challenge analysis - **Not Exploitable**: Theoretical bug with no practical trigger path --- ## Phase 6: Patch Development & Verification ### 6.1 Write the Patch **STOP — Confirm source version before writing ANY patch code:** ```bash echo -n "Patch base: " && git describe --tags HEAD LATEST=$(git describe --tags --abbrev=0 origin/master 2>/dev/null) echo "Latest upstream: $LATEST" BEHIND=$(git log --oneline HEAD..origin/master 2>/dev/null | wc -l) [ "$BEHIND" -gt 0 ] && echo "ERROR: $BEHIND commits behind — checkout $LATEST first!" && exit 1 # Final dedup check against subsystem trees (nf/nf-next/net/bpf/etc.) # A fix may have landed in the subsystem tree since Phase 2.2 SUBSYS_REMOTE="nf" # adjust per subsystem SUBSYS_NEXT="nf-next" # adjust per subsystem git fetch "$SUBSYS_REMOTE" 2>/dev/null; git fetch "$SUBSYS_NEXT" 2>/dev/null SEARCH_REFS="origin/master" git rev-parse --verify "$SUBSYS_REMOTE/main" &>/dev/null && SEARCH_REFS="$SEARCH_REFS $SUBSYS_REMOTE/main" git rev-parse --verify "$SUBSYS_NEXT/main" &>/dev/null && SEARCH_REFS="$SEARCH_REFS $SUBSYS_NEXT/main" echo "Dedup check against: $SEARCH_REFS" git log $SEARCH_REFS --oneline -S '' -- "$VFILE" | head -3 # If output is non-empty → someone already fixed this. STOP and verify the existing fix. echo "OK — source is current, no existing fix found, safe to write patch" ``` **If HEAD is behind origin/master → go back to Phase 2.2 and update the tree.** **If the dedup check finds a fix in the subsystem tree → STOP, report the existing fix.** Do NOT write a patch against stale source or duplicate existing work. This check takes a few seconds and prevents rejected patches and wrong report metadata. Read `references/patch-writing-guide.md` for Linux kernel patch conventions. **Fix Scope Decision (classify the bug before writing any code)** The correct *shape* of a fix depends on the bug class. Pick one — getting this wrong is exactly how a validation-layer patch earns the "you spot-fixed one check and moved the rest into a helper — that's not comprehensive verification" rejection. - **Point fix** — a specific lifetime, locking, refcount, or logic error in *trusted internal* code. → **Minimal, targeted diff.** Fix exactly the invariant violation. Prove each guard is reachable (Phase 3 Step B.1). Do NOT add symmetric or defensive checks elsewhere without independently proving reachability. - **Validation-layer hardening** — a bug in code that parses/decodes/deserializes *untrusted or on-disk data* (see Phase 3.5). → **Comprehensive, layered verification.** The one check the crash exposed is not the fix. Deliver the two-tier design from Phase 3.5 (generic structural + per-type). Here "minimal diff" means *no unrelated refactoring* — NOT "one check": every malformation class the untrusted input allows must be covered, because each is a real attack path. Reachability is satisfied by construction (the input is attacker- controlled); the only checks to omit are for states a caller already fully guarantees. When unsure which mode applies, ask: *is this code's contract to produce well-formed structures from arbitrary input?* If yes → hardening mode. **Design before code (required for anything beyond a one-line fix)** Work the fix at the high level *first*, in prose, and confirm it before generating code. This is the step that most improves patch quality and is the one most often skipped — do not drop to "code level" until the design is settled. 1. State the fix architecture in natural language: what invariant you enforce, where, and why that is the correct layer. For hardening bugs, this is the two-tier verification design from Phase 3.5. 2. For any fix that touches more than one site or introduces a new abstraction, decompose into a **single-change-per-patch series** — each patch does exactly one thing and is independently reviewable and bisectable (e.g. 1: add the generic verifier; 2: wire it in at the commit point; 3: add per-type verify callbacks; 4: enforce them before commit). List the series before writing any hunk. 3. Present the architecture + patch breakdown to the user and get agreement before coding. Then write the code one patch at a time. **Core principles**: - Fix the root cause, not the symptom (don't just add a NULL check if the real bug is a missing lock) - Minimal diff — no unrelated refactoring or style churn. "Minimal" bounds *scope*, not *thoroughness*: for validation-layer bugs (see Fix Scope Decision above) the necessary change covers every malformation class of the untrusted input, not just the crash path. - Follow the existing code style of the file you're modifying - **Comments only when necessary** (非必要不要注释): match the file's existing comment density; never comment *what* obvious code does, only the non-obvious *why* (subtle locking, a spec/hardware quirk, an inferrable-nowhere invariant). Line-by-line narration is an AI tell and gets rejected as noise. See `references/patch-writing-guide.md` § "Comments: Only When Necessary". - Add appropriate locking, reference counting, or lifetime management - Consider all callers — your fix must not break other code paths - **Verify each fix point is necessary**: For every guard you add, prove the bad state can actually reach that site (see Phase 3, Step B.1). For *point fixes*, do NOT add symmetric or defensive fixes "for completeness" without proving reachability independently — that wastes reviewer time and signals shallow analysis. For *validation-layer hardening*, covering every way the untrusted input can be malformed IS necessary (each is a real attack path) and is not "defensive completeness" — omit only checks for states a caller already fully guarantees. **Commit message format**: The commit message MUST include the decoded backtrace and follow upstream tag conventions. Look at any KASAN/bug fix in mainline `git log --grep='KASAN'` for real examples. **Write it concisely — no AI tells.** Terse, factual, human. Imperative subject with no "This patch..." preamble; explain the bug and the *why*, don't narrate the diff; cut filler ("In order to", "It is worth noting", "Additionally") and qualitative fluff ("robust", "comprehensive", "properly", "gracefully"); prose paragraphs, not bullet lists; don't hedge. If it reads like a blog post, cut it. See `references/patch-writing-guide.md` § "Write Concisely — Avoid AI Tells" for before/after examples. ``` subsystem: brief description of the fix Longer explanation of what the bug is, how it manifests, and why this patch fixes it. Use present tense. Include the root cause. Weave in the trigger conditions naturally — what config is needed, what privilege, whether it's reachable remotely or locally — so reviewers and stable maintainers can assess severity from the description itself. Introduce the backtrace naturally (upstream convention): syzbot reported a null-ptr-deref in icmp_unreach [1]: BUG: KASAN: null-ptr-deref in icmp_unreach (net/ipv4/icmp.c:1085) Call Trace: icmp_unreach (net/ipv4/icmp.c:1143) icmp_rcv (net/ipv4/icmp.c:1524) ip_protocol_deliver_rcu (net/ipv4/ip_input.c:205) ip_local_deliver_finish (net/ipv4/ip_input.c:234) ip_local_deliver (net/ipv4/ip_input.c:254) ip_rcv (net/ipv4/ip_input.c:569) The root cause is that icmp_tag_validation() dereferences inet_protos[proto] without checking for NULL... [1] https://syzkaller.appspot.com/bug?extid= Fixes: <12-char-hash> ("original commit title that introduced the bug") Reported-by: Closes: Link: Reviewed-by: Cc: stable@vger.kernel.org Signed-off-by: ``` **Commit message tag rules** (order matters): | Tag | Required? | Purpose | |---|---|---| | `Fixes:` | Yes | 12-char hash of the introducing commit | | `Reported-by:` | Yes | Who found the bug | | `Closes:` | Yes (if URL exists) | URL of the bug report — **required after `Reported-by:` per upstream convention** | | `Link:` | Recommended | lore.kernel.org link to the mailing list discussion | | `Tested-by:` | Recommended | Who tested the patch (can be `syzbot+@...` if syzbot tested it) | | `Reviewed-by:` | If reviewed | Code reviewer's signoff | | `Acked-by:` | If acked | Subsystem maintainer acknowledgment | | `Cc: stable@vger.kernel.org` | If applicable | Request backport to stable trees | | `Signed-off-by:` | Yes (last) | Developer Certificate of Origin | **Trigger conditions** (convey naturally in the commit body, not as a structured list): The commit message should make clear what's needed to trigger the bug — required CONFIG options, privilege level, whether it's local or remote, and whether the default config is affected. This helps reviewers and stable maintainers assess severity and backport priority. Weave this information into the natural prose description of the bug, the way upstream commits do. Do NOT use a bullet-point "Trigger conditions:" section — that format is not used upstream. Good (natural prose): ``` The bug is reachable from an unprivileged user namespace on any kernel with CONFIG_NETFILTER=y (default). A crafted nfnetlink message triggers a use-after-free in nf_tables_newrule(). ``` Bad (structured list — not upstream style): ``` Trigger conditions: - Required CONFIG: CONFIG_NETFILTER=y - Required privilege: unprivileged via userns - Attack vector: local (nfnetlink) ``` **Backtrace guidelines**: - Introduce naturally: `"syzbot reported a in [1]:"` or just paste the first crash line, NOT `"Decoded backtrace:"` (not used upstream) - Use the DECODED trace (file:line), not raw hex addresses - Trim to key frames: crash point + 5-10 relevant frames - Remove noise: timestamps, registers, `?` frames, `Code:` lines, module lists - Indent with single space - If from syzbot, add footnote `[1]` linking to the syzbot bug page ### 6.2 Validate the Patch After `git format-patch` produces the `.patch` file, run ALL of these checks. Do NOT skip any — a patch that fails checkpatch or doesn't apply cleanly will be rejected by maintainers without review. **Format validation**: ```bash # 1. checkpatch — coding style and commit message format ./scripts/checkpatch.pl --strict 0001-*.patch # Fix ALL errors and warnings. Common issues: # - Line over 80/100 characters # - Missing Signed-off-by # - Trailing whitespace # - Wrong indentation (spaces instead of tabs) # 2. Patch applicability — verify it applies cleanly to the tree git stash # save current state git am --check 0001-*.patch # dry-run: does it apply? # If "patch does not apply" → the patch context is wrong (wrong base commit) git stash pop # 3. Commit message format check head -1 0001-*.patch | grep '^From ' # Must start with "From " sed -n '4p' 0001-*.patch # Subject line # Verify: "Subject: [PATCH ] subsystem: description" # Subject should be < 72 characters (after the [PATCH ...] prefix) # Check body line length (should be < 75 chars, excluding backtrace) awk '/^---$/{exit} /^$/{next} length>75{print NR": "length" chars: "$0}' 0001-*.patch # 4. No MIME headers grep -E 'MIME-Version|Content-Type|Content-Transfer-Encoding' 0001-*.patch # Should find nothing. If present: commit message has non-ASCII, fix and regenerate. # 5. Verify required tags are present grep -E '^Fixes:|^Signed-off-by:|^Cc:' 0001-*.patch # Must have at least Fixes: and Signed-off-by: ``` **Build validation**: ```bash # 6. Compile test — apply patch and build git am 0001-*.patch make -j$(nproc) # Must compile without errors # 7. Sparse (optional but recommended) make C=1 # Static analysis for annotation issues # 8. Subsystem selftests (if they exist) make -C tools/testing/selftests/ run_tests ``` **Submission preparation**: ```bash # 9. Generate git send-email command with correct recipients ./scripts/get_maintainer.pl 0001-*.patch ``` **Patch generation workflow** — the commit message and code change must be bundled together via `git commit` + `git format-patch`. Do NOT use `git diff > patch` — that produces a raw diff without the commit message, tags, or authorship info. ```bash # Step 1: Stage the fix git add # Step 2: Commit with the full message (body + backtrace + tags) # Use a heredoc to preserve formatting git commit -m "$(cat <<'COMMIT_EOF' subsystem: brief description of the fix Root cause explanation — naturally mention what config, privilege, and attack vector are needed to trigger the bug, so reviewers can assess severity from the description. syzbot reported a in [1]: [1] https://syzkaller.appspot.com/bug?extid=... Fixes: <12-char-hash> ("introducing commit title") Reported-by: Closes: Cc: stable@vger.kernel.org Signed-off-by: COMMIT_EOF )" # Step 3: Generate the patch file with correct subject prefix git format-patch -1 --subject-prefix="PATCH " # This produces: 0001-subsystem-brief-description.patch # The file contains: From/Date/Subject headers + commit message + diff # Step 4: Verify the patch head -30 0001-*.patch # Should show From, Date, Subject, blank line, body grep -E 'MIME-Version|Content-Type' 0001-*.patch # Should find nothing # Step 5: Get maintainers and build the send command ./scripts/get_maintainer.pl 0001-*.patch # Auto-generate git send-email with correct recipients git send-email \ $(./scripts/get_maintainer.pl --nogit --nogit-fallback --norolestats \ 0001-*.patch | awk '{printf "--cc=\047%s\047 ", $0}') \ 0001-*.patch ``` Read `references/patch-writing-guide.md` § "Find Maintainers and Generate git send-email Command" for the full workflow including `tocmd`/`cccmd` auto-configuration. ### 6.3 Verify in QEMU — Two-Phase PoC Test (Required) Both phases are mandatory. You need to prove: (A) the bug exists, (B) your patch fixes it. Save the QEMU serial output from both runs as log files for the report. **Phase A: Confirm crash on VULNERABLE kernel** ```bash # Build vulnerable kernel (patch NOT applied) git stash # or checkout the un-patched commit make -j$(nproc) bzImage cp arch/x86/boot/bzImage report/kernel/test-bzImage cp vmlinux report/kernel/test-vmlinux # Boot and run PoC — capture output timeout 60 qemu-system-x86_64 \ -kernel report/kernel/test-bzImage \ -initrd rootfs.cpio.gz \ -append "console=ttyS0 root=/dev/ram rdinit=/init nokaslr oops=panic panic=1" \ -nographic -m 2G -smp 2 -no-reboot \ 2>&1 | tee report/logs/vuln-test.log # Verify the crash happened: grep -E 'BUG|KASAN|panic|Oops|NULL pointer' report/logs/vuln-test.log # MUST find crash indicators. If not → PoC doesn't trigger, fix the PoC first. ``` **Phase B: Confirm NO crash on PATCHED kernel** ```bash # Build patched kernel git stash pop # or apply the patch make -j$(nproc) bzImage cp arch/x86/boot/bzImage report/kernel/patched-bzImage cp vmlinux report/kernel/patched-vmlinux # Boot and run PoC — capture output timeout 60 qemu-system-x86_64 \ -kernel report/kernel/patched-bzImage \ -initrd rootfs.cpio.gz \ -append "console=ttyS0 root=/dev/ram rdinit=/init nokaslr oops=panic panic=1" \ -nographic -m 2G -smp 2 -no-reboot \ 2>&1 | tee report/logs/patched-test.log # Verify NO crash: grep -E 'BUG|KASAN|panic|Oops|NULL pointer' report/logs/patched-test.log # MUST find nothing. If crash indicators appear → patch is wrong, go back to 6.1. # Also check for new warnings introduced by the patch: grep -iE 'WARNING|lockdep|RCU' report/logs/patched-test.log # Any new warnings → investigate. Your patch may have introduced a regression. ``` **What to include in the report**: - `logs/vuln-test.log` — full serial output from Phase A (shows crash) - `logs/patched-test.log` — full serial output from Phase B (shows clean run) - Summary: "Vulnerable kernel: CRASH at . Patched kernel: PoC ran, no crash, clean dmesg." **For race condition bugs**: Run the PoC in a loop on the patched kernel to verify the fix is robust, not just lucky timing: ```bash # Inside QEMU on patched kernel: for i in $(seq 1 100); do ./poc; echo "Run $i: exit=$?"; done # All 100 runs should complete without crash ``` **Phase C: Unprivileged Trigger Test via `unshare` (if applicable)** If the Phase 5.2 permission gate analysis determined that ALL capability checks in the call path use `ns_capable()` (not `capable()`), the bug is reachable from an unprivileged user via `unshare(CLONE_NEWUSER|CLONE_NEWNET)`. In this case, you MUST verify this claim by actually running the PoC as a non-root user inside QEMU. **When to run Phase C**: Only when the permission gate analysis concludes "unprivileged via userns" — i.e., no `capable()` check blocks the path. ```bash # Inside QEMU (on the VULNERABLE kernel), create a non-root user and test: # 1. Create unprivileged user adduser --disabled-password --gecos "" testuser 2>/dev/null || true # 2. Copy PoC to user's home cp /root/poc /home/testuser/poc chmod +x /home/testuser/poc # 3. Run PoC as unprivileged user with unshare su testuser -c 'unshare -Urn /home/testuser/poc' # ^^^^ # -U = CLONE_NEWUSER (new user namespace, get all caps) # -r = map current user to root in new userns # -n = CLONE_NEWNET (new network namespace) # 4. Check result dmesg | tail -20 # If kernel crashes → confirmed: unprivileged trigger via userns # If EPERM or no crash → the permission analysis was wrong, re-check ``` **If the unprivileged trigger succeeds**: This is a significant finding. The report MUST include a dedicated section documenting this: ```markdown ### Unprivileged Trigger Verification The bug can be triggered by an unprivileged user via user namespaces: - **Test**: PoC run as non-root user with `unshare -Urn ./poc` - **Result**: Kernel crash confirmed — no root privileges required - **Permission path**: All checks use `ns_capable()` / `netlink_capable()` → `unshare(CLONE_NEWUSER|CLONE_NEWNET)` grants sufficient capabilities - **Impact upgrade**: This is NOT a "requires CAP_NET_ADMIN" bug — it is an unprivileged local DoS/exploit on any system with CONFIG_USER_NS=y and unprivileged_userns_clone=1 (Ubuntu default) Tested command: `su testuser -c 'unshare -Urn /home/testuser/poc'` Log: logs/unpriv-test.log ``` **If the unprivileged trigger fails** (EPERM): Document that too — it means the permission analysis missed a `capable()` check somewhere. Go back to Phase 5.2 and find the missing gate. **Save the log**: `report/logs/unpriv-test.log` ### 6.4 Regression Check - Ensure no new KASAN/UBSAN warnings in `patched-test.log` - If the subsystem has selftests, run them on the patched kernel - Test edge cases: what happens at the boundaries of your fix? - For race condition fixes: verify with `-smp 4` and stress testing ### 6.5 Restore Source Tree (Required) After verification is complete, the kernel source tree MUST be restored to its original (unpatched) state. The analysis process must be non-destructive — the patch lives only in the report's `patch/` directory as a standalone `.patch` file. ```bash # Restore all modified source files git checkout -- # Verify clean state git diff --stat # should show no changes ``` The correct workflow for the entire build-and-test cycle: ``` 1. Save the patch: git diff > report/patch/0001-fix.patch 2. Stash changes: git stash 3. Build VULNERABLE kernel from clean source 4. Copy vulnerable bzImage/vmlinux to report/kernel/test-* 5. Pop stash: git stash pop 6. Build PATCHED kernel 7. Copy patched bzImage/vmlinux to report/kernel/patched-* 8. Test both in QEMU 9. RESTORE source: git checkout -- ← DO NOT FORGET ``` If you skip step 9, the source tree is left in a modified state, which is messy and may confuse subsequent analyses or the user's own work. --- ## GATE CHECK: Before Entering Phase 7 — Evidence Required **DO NOT proceed to Phase 7 until you have ACTUAL QEMU OUTPUT as evidence.** A mental checklist is not enough. You must have run real commands and captured real output. The gate check is not "did you think about testing" — it is "show me the QEMU log." ### Required Evidence (must exist before report generation starts) Before writing any report file, you must have ALL of these **actual artifacts**: **1. Vulnerable kernel crash log** (`logs/vuln-test.log`) - Must contain actual QEMU serial output from booting the vulnerable kernel - Must show the PoC running and the kernel crashing - Verify: `grep -E 'BUG|KASAN|panic|Oops' logs/vuln-test.log` returns matches **2. Patched kernel clean log** (`logs/patched-test.log`) - Must contain actual QEMU serial output from booting the patched kernel - Must show the PoC running and the kernel NOT crashing - Verify: `grep -E 'BUG|KASAN|panic|Oops' logs/patched-test.log` returns NOTHING **3. Patched kernel binaries** (`kernel/patched-bzImage`, `kernel/patched-vmlinux`) - Must be actual compiled kernel binaries, not empty files - Verify: `file kernel/patched-bzImage` shows "Linux kernel x86 boot executable" **4. checkpatch output** - Must have run `./scripts/checkpatch.pl --strict` on the `.patch` file - No errors allowed (warnings may be acceptable with justification) ### How to verify the gate (run these commands) ```bash # All 4 checks must pass: echo "=== Gate Check ===" echo -n "1. Vuln crash log: " grep -cE 'BUG|KASAN|panic|Oops' report/logs/vuln-test.log 2>/dev/null && echo "PASS" || echo "FAIL — no crash log" echo -n "2. Patched clean log: " [ -f report/logs/patched-test.log ] && \ ! grep -qE 'BUG|KASAN|panic|Oops' report/logs/patched-test.log && echo "PASS" || echo "FAIL — crash on patched kernel or no log" echo -n "3. Patched kernel binary: " file report/kernel/patched-bzImage 2>/dev/null | grep -q 'Linux kernel' && echo "PASS" || echo "FAIL — no patched kernel" echo -n "4. Source restored: " git diff --stat 2>/dev/null | grep -q '.' && echo "FAIL — uncommitted changes" || echo "PASS" echo -n "5. Unpriv test (if ns_capable path): " if [ -f report/logs/unpriv-test.log ]; then echo "PASS — log exists" else echo "SKIP (not applicable, or FAIL if ns_capable path was identified)" fi ``` **If checks 1-4 show FAIL → you cannot proceed to Phase 7. Go back and fix it.** **Check 5 is required only when Phase 5.2 identified an `ns_capable()` path — if so, FAIL = go back.** ### What happens if QEMU is unavailable If `qemu-system-x86_64` is not installed or the user's environment doesn't have it: 1. **Ask the user**: "I need to verify the patch in QEMU. Where is your test environment?" 2. **Check for existing QEMU/rootfs**: `which qemu-system-x86_64`, `ls *.cpio.gz` 3. **Build one**: Run `scripts/setup_qemu_env.sh` to create the environment 4. **If truly impossible**: Tell the user explicitly: "I was unable to verify the patch in QEMU. The report is UNVERIFIED." Mark the report with a prominent warning banner. **NEVER silently skip testing and deliver a report as if it were verified.** ### Common excuses for skipping verification (all invalid) | Excuse | Why it's wrong | |---|---| | "The patch is obviously correct" | Obvious patches have hidden bugs | | "The prove log already showed it crashes" | That tested the VULNERABLE kernel, not the PATCHED one | | "I'll test later" | The user will trust the report as-is | | "QEMU isn't available" | Ask the user. Build one. Don't skip. | | "It's just a NULL check" | Even a NULL check can be wrong scope/variable/return | | "I already know it works" | You don't know until you see QEMU output with zero crash indicators | --- ## Phase 7: Report Generation & Artifact Packaging ### 7.1 Output Directory Structure Create a self-contained analysis folder. **Generate two separate reports** — one in English (`report_en.md`) and one in Chinese (`report_cn.md`). Each report is a complete standalone document (not a translation stub that references the other). Technical content (code snippets, diffs, struct layouts, ASCII diagrams, shell commands) should be identical in both; only the prose (explanations, analysis narrative, table headers) differs by language. ``` -analysis/ ├── report_en.md # Full analysis report (English) ├── report_cn.md # Full analysis report (Chinese / 中文) ├── poc/ │ ├── poc.c # PoC source code │ ├── Makefile # PoC build instructions │ └── poc_binary # Compiled PoC (optional) ├── patch/ │ ├── 0001-fix-description.patch # The git format-patch output │ └── patch_description.md # Patch explanation ├── kernel/ │ ├── test-vmlinux # Vulnerable kernel vmlinux (for GDB) │ ├── test-bzImage # Vulnerable kernel bzImage │ ├── patched-vmlinux # Patched kernel vmlinux │ ├── patched-bzImage # Patched kernel bzImage │ └── .config # Kernel config used ├── env/ │ ├── rootfs.cpio.gz # Root filesystem image │ ├── run-vulnerable.sh # Script to boot vulnerable kernel │ └── run-patched.sh # Script to boot patched kernel └── logs/ ├── crash.log # Original crash log ├── gdb-session.log # Key GDB session output └── patch-verification.log # Log showing PoC no longer crashes ``` ### 7.2 Report Templates Two templates are provided: - `assets/report_template.md` — English template - `assets/report_template_cn.md` — Chinese (中文) template Generate **both** `report_en.md` and `report_cn.md` using their respective templates. Do NOT create a single bilingual report with interleaved EN/CN sections — each report must be a complete, self-contained document in one language. Both reports must include: 1. **Executive Summary**: One-paragraph overview — what the bug is, severity, exploitability, fix status 2. **Bug Classification**: Type, affected subsystem, affected versions, CVE (if assigned) 3. **Crash Log Analysis**: Annotated crash log with key signals highlighted 4. **Root Cause Analysis**: Detailed technical analysis with code references 5. **Vulnerability Timeline**: Object lifecycle, race window, or error path that leads to the bug 6. **Exploitability Assessment**: Rating and justification, with references to similar known exploits 7. **Patch**: The fix, with explanation of why it's correct 8. **Verification Results**: Evidence that the patch fixes the bug without regressions 9. **References**: Related CVEs, kernel commits, kernelctf entries ### 7.3 Run Scripts Generate `run-vulnerable.sh` and `run-patched.sh` that boot QEMU with the correct parameters: ```bash #!/bin/bash # run-vulnerable.sh SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" qemu-system-x86_64 \ -kernel "$SCRIPT_DIR/../kernel/test-bzImage" \ -initrd "$SCRIPT_DIR/rootfs.cpio.gz" \ -append "console=ttyS0 root=/dev/ram rdinit=/init nokaslr" \ -nographic -m 2G -smp 2 \ -net nic -net user,hostfwd=tcp::10022-:22 ``` --- ## Subagent Orchestration Patterns ### Pattern 1: Parallel Static Analysis ``` Main Agent (Coordinator): ├── Subagent A: Read and analyze functions in the crash call stack ├── Subagent B: Git log/blame the affected files, find introducing commit ├── Subagent C: Search knowledge base for similar vulnerability patterns └── Main Agent: Synthesize results → determine root cause hypothesis ``` ### Pattern 2: Build & Test Pipeline ``` Main Agent (Coordinator): ├── Subagent D [worktree]: Build vulnerable kernel with debug configs ├── Subagent E [worktree]: Build patched kernel └── Main Agent: When both complete → verify with QEMU ``` ### Pattern 3: Cross-Reference Research ``` Main Agent (Coordinator): ├── Subagent F: Search kernelctf for exploits in same subsystem ├── Subagent G: Search for related CVEs and patches upstream ├── Subagent H: Analyze slab cache sharing for exploitability └── Main Agent: Synthesize exploitability assessment ``` ### When to Use Subagents vs. Do It Yourself - **Use subagents** when: tasks are independent, IO-bound (file reading, git operations, web fetch), or you need isolated worktrees (compilation) - **Do it yourself** when: the task is a quick single-file read, or results of step N are needed for step N+1 with no parallelism opportunity --- ## Important Considerations ### Security & Ethics This skill is for **authorized security research, defensive analysis, and educational purposes**. The goal is to understand vulnerabilities to fix them and improve kernel security. - Always focus on developing correct patches, not weaponizing bugs - If a PoC is needed for verification, keep it minimal — just enough to trigger the bug - Report findings responsibly ### Linux Coding Standards When writing patches, strictly follow: - `Documentation/process/coding-style.rst` - `Documentation/process/submitting-patches.rst` - Run `scripts/checkpatch.pl --strict` on all patches - Tabs for indentation, 80-column lines (soft limit, 100 hard) - No unnecessary whitespace changes - Commit messages: imperative mood, 72-char subject, wrapped body ### Avoiding Patch Regressions - Test with multiple kernel configs (defconfig, allyesconfig with relevant options) - Consider all callers of modified functions - If adding locking, verify no deadlock potential (lock ordering) - If modifying reference counting, verify no leaks or double-frees introduced - If changing error paths, verify all resources are properly cleaned up ### Tools Reference | Tool | Purpose | |---|---| | `scripts/decode_stacktrace.sh` | Decode raw kernel stack traces | | `scripts/get_maintainer.pl` | Find subsystem maintainers | | `scripts/checkpatch.pl` | Validate patch coding style | | `virtme-ng (vng)` | Quick kernel boot for testing | | `addr2line` | Resolve addresses to source lines | | `pahole` | Inspect struct layouts and padding | | `crash` | Kernel crash dump analysis | | `ftrace` | In-kernel function tracer (no recompile needed) | | `perf` | Hardware performance counters, timing analysis | | `objdump` | Disassembly when source-level debugging isn't enough | | `strace` | Trace PoC syscall sequences | | `slabinfo` | Runtime slab cache analysis | --- ## Reference Files These files contain detailed guidance for specific phases. Read them when you reach the relevant phase — they contain information too detailed for the main skill but essential for correct analysis. - `references/crash-log-analysis.md` — How to parse KASAN, UBSAN, BUG, panic logs with examples - `references/vuln-classification.md` — Taxonomy of kernel vulnerability classes and identification - `references/exploitability-assessment.md` — Framework for assessing kernel bug exploitability - `references/patch-writing-guide.md` — Linux kernel patch conventions and common patterns - `references/verification-hardening.md` — Comprehensive verification analysis for parsers/decoders/deserializers of untrusted input (two-tier design, trust-boundary placement, single-change-per-patch decomposition) - `references/qemu-setup.md` — Setting up QEMU+GDB kernel debugging environments - `references/kernelctf-knowledge-base.md` — Exploit techniques and patterns from Google's kernelCTF - `references/syzbot-workflow.md` — Syzbot interaction: reproducers, `#syz test`, `#syz fix` - `references/regression-testing.md` — Kselftest, KUnit, LTP, multi-config testing - `assets/report_template.md` — Template for the final analysis report ### Quick Reference Index | When you encounter... | Read this | |---|---| | KASAN crash log | `references/crash-log-analysis.md` § KASAN Reports | | syzbot report URL | `references/syzbot-workflow.md` § Reading a Syzbot Report | | UAF / double-free / OOB | `references/vuln-classification.md` | | Bug in a parser / decoder / log-recovery / deserializer | `references/verification-hardening.md` + SKILL.md Phase 3.5 | | "Is this exploitable?" | `references/exploitability-assessment.md` § Assessment Methodology | | Writing a patch | `references/patch-writing-guide.md` § Common Fix Patterns | | Setting up QEMU+GDB | `references/qemu-setup.md` § Full QEMU Setup | | Similar known exploits | `references/kernelctf-knowledge-base.md` § Exploit Technique Catalog | | Writing regression test | `references/regression-testing.md` § Writing a New Selftest | | Submitting patch upstream | `references/patch-writing-guide.md` § Submitting Patches Upstream | | Defensive blocking | `references/exploitability-assessment.md` § Syscalls to Block |