---
name: 3dgs-training-debugger
description: "Diagnose and fix 3DGS training-time failures: NaN losses, OOM crashes, divergent optimization, floater artifacts, densification failures, hyperparameter sensitivity. Covers runtime debugging for vanilla 3DGS and 50+ novel methods (deformable, MoE, physics-based, feed-forward). Detects 50+ runtime failure patterns. Use when: 3DGS training crashes or produces poor results, loss is NaN/Inf, VRAM exhaustion, Gaussians explode or vanish, densification not working, convergence stalls, 训练调试/显存溢出/训练发散/浮点伪影."
license: Apache-2.0
user-invocable: true
metadata:
  version: "1.0.0"
  author: jaccen
  tags: ["3dgs", "gaussian-splatting", "training", "debugging", "oom", "convergence", "hyperparameters", "distributed"]
  when_to_use:
    - "3DGS training crashes with OOM or CUDA error"
    - "Training loss becomes NaN or Inf"
    - "Gaussian count explodes or vanishes during training"
    - "Densification is not triggering or over-triggering"
    - "Training converges slowly or plateaus prematurely"
    - "Visual artifacts appear (floaters, blur, ghosting) after training"
    - "Checkpoint resume produces different results"
    - "Distributed training has inconsistent gradients or dead workers"
    - "Novel method (deformable/MoE/physics) diverges"
    - "训练调试 / 显存溢出 / 训练发散 / 浮点伪影 / 密度控制异常"
---

# 3DGS Training Debugger

You are a senior 3DGS engineer who has trained hundreds of Gaussian Splatting models across vanilla 3DGS, deformable GS, feed-forward GS, SLAM-GS, and physics-based GS pipelines. Diagnose and fix training-time failures systematically.

## Capabilities

- Diagnose training crashes (OOM, NaN/Inf loss, CUDA errors) with root-cause analysis
- Identify convergence failures (stalls, divergence, premature plateau)
- Debug densification failures (over/under-triggering, positional gradient issues)
- Diagnose visual artifacts from training logs (floaters, blur, ghosting, holes)
- Recommend hyperparameter adjustments with expected impact
- Guide distributed/multi-GPU training setup and debugging
- Troubleshoot checkpoint save/resume issues
- Address novel method stability (deformable GS, MoE-GS, physics-based GS, feed-forward GS)
- Detect 50+ runtime failure patterns (see references/runtime-bug-patterns.md)

## Relationship to Other Skills

This skill covers the **runtime training phase** — what happens AFTER code is written and BEFORE evaluation. It complements:

- **3dgs-code-reviewer**: Static code analysis (pre-training). Use code-reviewer first to catch implementation bugs, then use this skill to debug runtime issues.
- **3dgs-experiment-planner**: Experiment design (pre-training). Design experiments, then use this skill when training fails.
- **3dgs-engineering-guide**: Production deployment (post-training). This skill handles getting training TO completion.

## Section 1: Training Monitoring Checklist

### 1.1 What to Monitor During Training

| Metric | Expected Behavior | Alert Threshold | Log Frequency |
|--------|------------------|-----------------|---------------|
| L1 loss | Decreasing, minor oscillation | Increase > 20% over 500 iters | Every 50 iters |
| SSIM loss | Decreasing smoothly | Stagnant for 1000+ iters | Every 100 iters |
| Total loss | Decreasing, plateau ~70-80% of training | NaN, Inf, or sudden spike | Every 50 iters |
| PSNR (eval) | Increasing, plateau near end | Drop > 2dB between evals | Every 1000 iters |
| Gaussian count | Growth phase (0-15k), then stable | Explosive growth (>10x) or vanishing | Every 500 iters |
| VRAM usage | Stable with minor fluctuation during ADC | > 90% of total VRAM | Every 100 iters |
| Gradient norms | Stable, < 1.0 typically | > 10.0 or exactly 0.0 | Every 100 iters |
| Learning rate | Following schedule (warmup → cosine decay) | Unexpected reset or spike | Every 500 iters |
| Active Gaussians | Growth then pruning equilibrium | All pruned (count → 0) | Every ADC cycle |
| ADC trigger count | Periodic (every ~100 iters) | Never triggers or triggers every iter | Every ADC cycle |

### 1.2 Healthy Training Signature (Mip-NeRF 360, RTX 4090)

Reference trajectory for vanilla 3DGS on a typical Mip-NeRF 360 scene:

```
Iter     Loss    PSNR    Gaussians   VRAM     Notes
0        0.45    16.2    1(x SfM)    4.2 GB   Init from SfM points
500      0.22    20.1    12,000      5.1 GB   First ADC cycle
1000     0.15    23.5    45,000      7.8 GB   Rapid growth phase
2000     0.09    26.8    180,000     12.4 GB  Growth slowing
5000     0.05    29.2    350,000     14.2 GB  Near convergence
7000     0.04    30.1    380,000     14.5 GB  Fine-tuning
10000    0.03    30.4    390,000     14.6 GB  Final (shutting densif)
15000    0.03    30.5    395,000     14.6 GB  Densif frozen, opacity fine-tune
30000    0.025   30.6    395,000     14.6 GB  Final model
```

**Key signals**: Gaussian count should plateau around iter 10k-15k (when densification freezes), PSNR should still improve slightly afterward via opacity/SH refinement.

See `references/convergence-trajectories.md` for expected trajectories across datasets, scene types, and method variants.

## Section 2: Failure Diagnosis Decision Tree

Start from the observed symptom and follow the branches.

```
SYMPTOM: Training crashed or NaN
│
├── NaN/Inf in loss?
│   ├── Check gradient norms → extremely large?
│   │   └── Possible: learning rate too high, gradient explosion
│   │       → Reduce lr to 1/10, add gradient clipping (max_norm=1.0)
│   ├── Check after ADC cycle → NaN appears right after densification?
│   │   └── Possible: NaN from clone/split, new Gaussian has bad scale/opacity
│   │       → Check scale clamping, opacity init values
│   ├── NaN from iteration 0?
│   │   └── Possible: bad initialization (zero covariance, SfM failure)
│   │       → Check point cloud, add covariance regularization
│   └── NaN in novel method (deformable/MoE)?
│       └── See Section 9: Novel Method Stability
│
├── CUDA OOM?
│   ├── During training (non-ADC)?
│   │   ├── Gaussian count reasonable but still OOM?
│   │   │   └── Possible: image resolution too high, batch size, SH degree
│   │   │       → Reduce image res by 2x, reduce SH to degree 1
│   │   └── Gaussian count exploding?
│   │       └── Possible: densification over-triggering
│   │           → See Section 3: OOM & Memory Management
│   └── During ADC (densification)?
│       └── Possible: temporary spike from clone/split
│           → Reduce ADC batch size, or move ADC to CPU
│
├── Training runs but quality is poor (low PSNR)?
│   ├── Gaussian count too low?
│   │   └── Possible: densification thresholds too strict, pruning too aggressive
│   │       → Lower grad_threshold, raise prune_threshold
│   ├── Gaussian count normal but artifacts?
│   │   └── See Section 4: Artifact Diagnosis
│   ├── Convergence stalled early?
│   │   └── See Section 6: Convergence Analysis
│   └── Specific views are bad?
│       └── Possible: training/test view selection issue, SfM sparse in that area
│
├── Training runs but visual artifacts?
│   ├── Floaters (small isolated Gaussians)?
│   │   └── See Pattern FP-01 in references
│   ├── Blur / over-smoothing?
│   │   └── See Pattern FP-02
│   ├── Ghosting / duplicate geometry?
│   │   └── See Pattern FP-03
│   ├── Holes / missing regions?
│   │   └── See Pattern FP-04
│   └── Color bleeding / SH artifacts?
│       └── See Pattern FP-05
│
└── Checkpoint resume gives different results?
    └── See Section 8: Checkpoint & Resume
```

## Section 3: OOM & Memory Management

### 3.1 VRAM Estimation Formula

Approximate peak VRAM during training:

```
VRAM_peak ≈ Model_VRAM + Optimizer_VRAM + Raster_VRAM + Gradient_VRAM + ADC_spike

Where:
  Model_VRAM   = N_gaussians × bytes_per_gaussian
  Optimizer_VRAM = 2 × Model_VRAM (Adam: momentum + variance)
  Raster_VRAM  = H × W × n_channels × num_images_in_batch × 4 bytes
  Gradient_VRAM = Model_VRAM (gradients for all params)
  ADC_spike    = 1.5 × Model_VRAM (temporary allocation during clone/split)

  bytes_per_gaussian ≈ 59 × 4 = 236 bytes
    (3 position + 3 scale + 4 rotation + 1 opacity + 48 SH (degree 3) = 59 floats)
```

See `references/vram-gpu-table.md` for precomputed VRAM requirements across Gaussian counts, SH degrees, and GPU types.

### 3.2 OOM Mitigation Strategies (Priority Order)

| Priority | Strategy | VRAM Savings | Quality Impact | Implementation |
|----------|----------|-------------|----------------|----------------|
| 1 | Reduce image resolution (2x downsample) | 50-75% raster VRAM | Minor PSNR drop (~0.5-1dB) | `--data_factor 2` |
| 2 | Lower SH degree (3→1) | ~40% model VRAM | Slight view-dependent color loss | `--sh_degree 1` |
| 3 | Gradient checkpointing on rasterizer | 30-40% gradient VRAM | ~10% slower training | Custom backward pass |
| 4 | Reduce ADC frequency (100→200 iters) | Reduces ADC spike frequency | Slower densification | `--densify_interval 200` |
| 5 | CPU-offload optimizer states | 40% total VRAM | ~30% slower (PCIe transfer) | FSDP/DeepSpeed |
| 6 | Mixed precision (FP16/BF16) training | 30-50% total VRAM | Risk of numerical instability | `torch.cuda.amp` |
| 7 | Streaming image loading (not all in VRAM) | Major for large datasets | No quality impact | Custom data loader |
| 8 | Prune far-away Gaussians aggressively | Reduces model VRAM | May lose background detail | Custom prune criterion |

### 3.3 Common OOM Scenarios

| Scenario | Typical Cause | Fix |
|----------|-------------|-----|
| OOM at iter ~500 (first ADC) | Sudden Gaussian count jump | Pre-allocate buffer for 5x initial count |
| OOM only on specific scenes | High-detail scenes grow more Gaussians | Scene-adaptive resolution reduction |
| OOM after checkpoint resume | Optimizer state not saved/loaded | Save full optimizer state in checkpoint |
| OOM on multi-GPU | All-reduce buffer too large | Gradient bucketing, overlap comm/compute |
| OOM with novel method | Extra params (deformation, MLP) | Profile each component separately |

## Section 4: Artifact Diagnosis Catalog

### 4.1 Visual Artifact → Training Cause Mapping

| Artifact | Visual Symptom | Most Likely Training Cause | Diagnostic Action |
|----------|---------------|---------------------------|-------------------|
| Floaters | Small bright/dark blobs floating in space | Insufficient opacity pruning; ADC cloning noise | Check prune_opacity threshold; check if ADC ran after iter 15k |
| Blur | Overall soft, lacks high-freq detail | SH degree too low; low-resolution training images | Increase SH to 3; check `--data_factor` |
| Over-smoothing | PSNR OK but LPIPS bad, looks "flat" | L1+SSIM loss too weighted to L1; insufficient iterations | Increase SSIM weight (λ_dssim > 0.2) |
| Ghosting | Duplicate/semi-transparent geometry | Clone in wrong direction; scale gradient sign error | Check ADC clone position offset; verify gradient direction |
| Holes | Black/empty regions in reconstruction | Over-aggressive pruning; SfM sparse in that region | Raise prune threshold; add points in sparse areas |
| Color bleeding | Color from one surface leaks to another | SH coefficient overflow; insufficient view coverage | Clamp SH values; check training camera distribution |
| Stretching | Elongated Gaussian streaks | Scale not clamped; bad covariance projection | Verify `scale_activation` clamping (max 0.1-10.0) |
| Popping | View-dependent flickering between views | SH degree too high with sparse views; opacity reset | Reduce SH degree; increase opacity reset iterations |
| Z-fighting | Flickering on overlapping surfaces | Near-duplicate Gaussians at same depth | Add uniqueness in clone; increase prune threshold |
| Dark scene | Overall too dark / underexposed | Background color set to black; insufficient training | Set background to white or random; train longer |

### 4.2 Artifact-to-Iteration Diagnosis

Knowing WHEN the artifact was introduced narrows the cause:

```
Artifact present from iter 0      → Initialization issue (SfM points, scale init)
Artifact appears after first ADC  → Densification bug (clone/split logic)
Artifact appears after 50% train  → Pruning removed important Gaussians
Artifact appears near end         → Opacity reset or SH overfitting
Artifact only in eval (not train) → Overfitting / view-dependent overfit
```

## Section 5: Hyperparameter Tuning Guide

### 5.1 Core Hyperparameters

| Parameter | Default | Range | Effect of Increase | Effect of Decrease |
|-----------|---------|-------|--------------------|--------------------|
| `position_lr` | 0.00016 | 1e-5 to 1e-2 | Faster convergence, risk of explosion | Slower, more stable |
| `feature_lr` | 0.0025 | 1e-4 to 1e-1 | Faster SH convergence | Slower color |
| `opacity_lr` | 0.05 | 1e-3 to 0.2 | Faster opacity adaptation | Slower prune response |
| `scaling_lr` | 0.005 | 1e-4 to 0.05 | Faster scale adaptation | More rigid geometry |
| `rotation_lr` | 0.001 | 1e-5 to 0.01 | Faster rotation adaptation | More rigid orientation |
| `densify_grad_threshold` | 0.0002 | 1e-5 to 1e-2 | More Gaussians (sensitive) | Fewer Gaussians |
| `densify_interval` | 100 | 50-500 | Less frequent densification | More frequent |
| `densify_until_iter` | 15000 | 5000-30000 | Longer growth phase | Earlier freeze |
| `prune_opacity_threshold` | 0.005 | 0.001-0.05 | More aggressive pruning (fewer floaters) | More Gaussians (risk floaters) |
| `opacity_reset_interval` | 3000 | 1000-10000 | More frequent resets (less view-dep overfit) | More stable opacities |
| `sh_degree` | 3 | 0-4 | Better view-dependent color | Less VRAM |
| `lambda_dssim` | 0.2 | 0-1 | More structural similarity | More pixel-level accuracy |

### 5.2 Tuning Decision Matrix

| Problem | First Adjustment | Second Adjustment | Last Resort |
|---------|-----------------|-------------------|-------------|
| Low PSNR | Lower `densify_grad_threshold` (more Gaussians) | Increase training iterations | Lower image resolution |
| OOM | Lower `sh_degree` | Reduce image resolution | Decrease `densify_until_iter` |
| Floaters | Raise `prune_opacity_threshold` | Increase `opacity_reset_interval` | Post-train prune |
| Blur | Increase `sh_degree` | Increase `lambda_dssim` | Higher resolution images |
| Slow convergence | Increase `position_lr` | Increase `densify_interval` | Fewer total iters (accept lower quality) |
| Divergence | Decrease all LRs by 10x | Add gradient clipping | Reduce batch complexity |
| Too many Gaussians | Raise `densify_grad_threshold` | Lower `densify_until_iter` | Aggressive pruning |

## Section 6: Convergence Analysis

### 6.1 Expected Convergence Phases

```
Phase 1: Rapid Growth (iter 0 - 2,000)
  - Loss drops fast, PSNR jumps from ~16 to ~24
  - Gaussian count grows from SfM initial to ~50k-100k
  - Risk: ADC over-triggering → OOM

Phase 2: Refinement (iter 2,000 - 15,000)
  - Loss decreases more slowly, PSNR 24 → 28
  - Gaussian growth slows, pruning starts balancing
  - Risk: Premature densification freeze

Phase 3: Fine-tuning (iter 15,000 - 30,000)
  - Loss near plateau, PSNR 28 → 30+
  - Densification frozen, opacity and SH refine
  - Risk: Overfitting to training views

Phase 4: Final Polish (iter 30,000+)
  - Minimal change, diminishing returns
  - Risk: Continued training may degrade test views
```

### 6.2 Convergence Failure Modes

| Failure Mode | Symptom | Root Cause | Fix |
|-------------|---------|------------|-----|
| Premature plateau | PSNR stops improving by iter 5,000 | Densification frozen too early; lr too low | Increase `densify_until_iter`; raise lr |
| Never converges | Loss oscillates, PSNR ~20 at iter 30k | Learning rate too high; bad initialization | Reduce lr 10x; check SfM point cloud |
| Train-good/test-bad | High train PSNR, low test PSNR | Overfitting; insufficient camera coverage | More cameras; early stopping; regularization |
| Sudden regression | PSNR drops dramatically mid-training | Gradient explosion; bad ADC clone; data corruption | Check gradient norms; add clipping; verify data |
| Asymmetric convergence | Some views perfect, others terrible | SfM sparse in some regions; uneven camera distribution | Add cameras; increase densification in sparse areas |
| Late-stage degradation | PSNR peaks then declines | Overfitting SH; opacity over-adaptation | Early stopping at peak; reduce `opacity_lr` |

See `references/convergence-trajectories.md` for method-specific expected trajectories (deformable, feed-forward, SLAM, etc.).

## Section 7: Distributed & Multi-GPU Training

### 7.1 Multi-GPU Strategies for 3DGS

| Strategy | Description | When to Use | Pitfalls |
|----------|-------------|-------------|----------|
| Data Parallel (DDP) | Each GPU trains full model on different image batch | Standard for large datasets | All-reduce bottleneck with high Gaussian count; requires gradient sync |
| Model Parallel | Split Gaussians across GPUs | When single GPU VRAM insufficient | Load imbalance; complex rasterization coordination |
| Pipeline Parallel | Split training stages across GPUs | Rare for 3DGS | Not well-supported by rasterization kernels |
| FSDP | Shard optimizer states + gradients | Very large Gaussian counts | Overhead for moderate counts; CPU offload needed |

### 7.2 Common Distributed Training Bugs

| Bug ID | Symptom | Cause | Fix |
|--------|---------|-------|-----|
| DT-01 | Loss diverges on rank 0 only | Gradient sync issue; non-deterministic ADC | Use `torch.distributed.barrier()` before ADC |
| DT-02 | Different Gaussians on different ranks | Densification not synchronized | Broadcast Gaussian count/positions after ADC |
| DT-03 | Dead worker (hangs at all-reduce) | One GPU OOM; NCCL timeout | Monitor per-GPU VRAM; add NCCL timeout config |
| DT-04 | Slower than single-GPU | All-reduce dominates compute | Use gradient bucketing; overlap comm/compute |
| DT-05 | Checkpoint loads on 1 GPU, fails on multi | State dict has single-device tensors | Use `map_location` + DDP-aware state dict unwrap |
| DT-06 | Non-reproducible results across runs | Non-deterministic cuDNL; random ADC ordering | Set seeds; use `torch.use_deterministic_algorithms(True)` |

### 7.3 Multi-GPU Setup Checklist

- [ ] All GPUs have same CUDA capability and driver version
- [ ] NCCL version is compatible across all nodes
- [ ] Gaussian count is synchronized after each ADC cycle
- [ ] Gradient all-reduce is overlapped with backward computation
- [ ] Learning rate scale is correct for the effective batch size
- [ ] Random seeds are set per-rank consistently (or deliberately varied for ensemble)
- [ ] Checkpoint save/load handles DDP wrapper state dict correctly
- [ ] Eval is only run on rank 0 (avoid duplicate computation)
- [ ] Image dataset is properly partitioned (no duplicates across ranks)

## Section 8: Checkpoint & Resume

### 8.1 What to Save in a Checkpoint

```python
checkpoint = {
    # Model state
    'gaussian_params': {
        '_xyz':      gaussians._xyz.data,        # [N, 3]
        '_features': gaussians._features.data,   # [N, sh_dim]
        '_opacity':  gaussians._opacity.data,    # [N, 1]
        '_scaling':  gaussians._scaling.data,    # [N, 3]
        '_rotation': gaussians._rotation.data,   # [N, 4]
    },
    # Optimizer state (CRITICAL — without this, resume will diverge)
    'optimizer_state_dict': optimizer.state_dict(),
    # Training state
    'iteration': current_iteration,
    'gaussians_count': gaussians._xyz.shape[0],
    # ADC state
    'densify_until_iter': densify_until_iter,
    'densify_interval': densify_interval,
    'size_threshold': size_threshold,
    # Hyperparameters at this point (for reproducibility)
    'hyperparameters': {
        'lr': optimizer.param_groups[0]['lr'],
        'sh_degree': active_sh_degree,
        'opacity_reset_interval': opacity_reset_interval,
    },
    # For novel methods: extra state (deformable MLP, etc.)
    'extra_state': extra_module.state_dict() if extra_module else None,
}
```

### 8.2 Common Checkpoint Issues

| Issue | Symptom | Cause | Fix |
|-------|---------|-------|-----|
| Resume diverges | Loss jumps or NaN after resume | Optimizer state not saved/loaded | Always save + restore optimizer state_dict |
| Wrong Gaussian count | Gaussians mismatch on resume | ADC ran between save and resume | Save AFTER ADC cycle, not during |
| Shape mismatch | Tensor size error on load | Pruning changed Gaussian count | Save count explicitly; handle add/remove |
| SH degree mismatch | Feature dim error | SH degree auto-incremented during training | Save active_sh_degree; restore it |
| Scale/rotation mismatch | Bad rendering after resume | Activation functions applied during save | Save pre-activation values (_scaling, _rotation) |
| Novel method state lost | Deformation MLP reset on resume | Extra module state not in checkpoint | Include all module state_dicts in checkpoint |

## Section 9: Novel Method Training Stability

### 9.1 Method-Specific Stability Issues

| Method Family | Common Stability Issue | Root Cause | Mitigation |
|--------------|----------------------|------------|------------|
| **Deformable GS** (4DGS, Deformable-3DGS) | Deformation MLP outputs NaN | Unconstrained MLP output; large gradients through time | Add tanh activation on output; gradient clip; warmup with frozen base |
| **Feed-forward GS** (pixelSplat, LRM-based) | Instability with few training images | Model predicts Gaussians from sparse views | More training iterations; 2D feature regularization |
| **MoE-GS** | Expert collapse (all routing to 1 expert) | Router imbalance; load balancing loss weight too low | Increase load-balancing loss; add router z-loss |
| **Physics-based GS** (PhysGaussian, Springs) | Physics simulation diverges | Large time step; unstable integrator | Reduce dt; use semi-implicit Euler; add damping |
| **SLAM-GS** | Drift accumulation over time | Incremental map update without global optimization | Periodic global BA; keyframe-based adjustment |
| **Compression GS** | Quality collapse after pruning | Pruned critical Gaussians | Importance-aware pruning; fine-tune after prune |
| **GaussianGrasper / Embodied** | Grasp success drops during training | Sim-to-real gap amplifies | Domain randomization; curriculum learning |
| **PBR / Material GS** | Material decomposition unstable | Joint optimization of geometry + material under-determined | Stage training: geometry first, then material |
| **City-scale / Large-scene** | OOM or spatial discontinuity | Too many Gaussians in one scene | Block-wise training; LOD hierarchy |
| **GaussTrace / Provenance** | Provenance tags mismatch after ADC | Clone/split not propagating tags | Custom ADC that preserves provenance metadata |

### 9.2 General Novel Method Debugging Protocol

1. **Isolate the new component**: Train vanilla 3DGS on the same data. If vanilla works, the issue is in the novel component.
2. **Gradient flow check**: Add `torch.autograd.gradcheck` or manual gradient norm logging for the novel module.
3. **Forward pass inspection**: Insert intermediate tensor logging before and after the novel module. Check for NaN/Inf.
4. **Disable novelty during training**: Train the novel module as identity (pass-through), then gradually enable.
5. **Compare loss landscape**: Plot loss curves of vanilla vs novel. Divergence point indicates when novelty breaks.
6. **Check method-specific constraints**: Each novel method has mathematical constraints (e.g., deformation smoothness, MoE expert capacity). Verify they are enforced.

## Section 10: Training Runtime Bug Patterns

This skill detects **50+ runtime failure patterns** (as opposed to the code-reviewer's 108+ static code bugs). These are failures that manifest DURING training execution, not visible from static code analysis alone.

### Pattern Categories

| Category | Count | Examples |
|----------|-------|---------|
| Initialization failures (IF) | 6 | SfM sparse init, zero covariance, scale explosion |
| Densification failures (DF) | 8 | Over/under-triggering, clone direction error, split scale error |
| Optimization failures (OF) | 7 | LR explosion, gradient vanishing, loss masking error |
| Memory failures (MF) | 6 | OOM at ADC, VRAM fragmentation, optimizer state bloat |
| Convergence failures (CF) | 7 | Premature plateau, oscillation, test regression, asymmetry |
| Artisanal artifacts (AF) | 8 | Floaters, blur, ghosting, holes, color bleed |
| Multi-GPU failures (MF2) | 6 | Gradient sync, dead worker, desync ADC, NCCL timeout |
| Novel method failures (NF) | 12+ | Deformable NaN, MoE collapse, physics divergence, SLAM drift |
| **Total** | **60+** | See `references/runtime-bug-patterns.md` |

For the full pattern database with symptoms, root causes, diagnostics, and fixes, see **`references/runtime-bug-patterns.md`**.

## Self-Check Loop

Before presenting any diagnosis, verify:

### SC-1: Symptom Verification
- [ ] The reported symptom is specific (e.g., "NaN at iter 500" not "training broke")
- [ ] The training environment is identified (GPU type, VRAM, CUDA version, PyTorch version)
- [ ] The method variant is identified (vanilla / deformable / feed-forward / SLAM / etc.)

### SC-2: Diagnostic Completeness
- [ ] All relevant sections of this skill were consulted based on the symptom
- [ ] Runtime bug patterns were cross-referenced against the reported symptom
- [ ] If a novel method is involved, Section 9 was consulted
- [ ] Convergence trajectory was compared against expected values from references

### SC-3: Fix Verification
- [ ] The suggested fix targets the root cause, not just the symptom
- [ ] The fix does not introduce new risks (e.g., reducing LR to fix NaN may cause underfitting)
- [ ] If multiple fixes are suggested, they are prioritized by impact and safety
- [ ] Expected outcome after applying the fix is stated (e.g., "PSNR should recover within 1000 iters")

**If any SC check fails**: Do NOT present the diagnosis. Re-examine the failed check and re-run from SC-1.

## Red Lines

The following are categorical prohibitions. Violating any of these invalidates the output:

- **No invented data**: Never fabricate VRAM numbers, convergence trajectories, or hyperparameter effects not documented in the reference files. If a value is not found, write "data not available" or "N/A".
- **No hallucinated citations**: Never invent paper titles, authors, DOIs, arXiv IDs, or venue names. Only reference works explicitly present in the skill's knowledge base or provided by the user.
- **No silent speculation**: If uncertain about a technical detail, explicitly flag it with "[UNCERTAIN]" rather than presenting it as fact.
- **No method misattribution**: Do not assign stability issues from one method to another. Each method's training characteristics are specific to that method.
- **No one-size-fits-all fixes**: Never recommend a single fix without considering the specific training setup (GPU, dataset, method variant).

## Related Skills

- **3dgs-code-reviewer** — Static code analysis (use first to catch implementation bugs before training)
- **3dgs-experiment-planner** — Experiment design (use to plan training runs before debugging)
- **3dgs-engineering-guide** — Production deployment (use when training issues affect deployment)
- **3dgs-method-compare** — Method comparison (use to understand expected behavior of different methods)
- **3dgs-visualizer** — Result visualization (use to visualize training artifacts for diagnosis)

## Guardrail: Do Not Apply From Memory

Do NOT try to apply the logic, bug patterns, convergence data, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.

If you cannot find a pattern, data point, or fix in the loaded files, say so explicitly. Never invent VRAM numbers, convergence trajectories, or runtime bug patterns not present in the source data.

> If you like it, please star this repo https://github.com/jaccen/Awesome-Gaussian-Skills