--- name: owasp-llm-top10 description: OWASP Top 10 for LLM applications — full checklist for AI agent systems. Prompt injection, insecure output, training data poisoning, DoS, supply chain, sensitive info disclosure, insecure plugins, excessive agency, overreliance, and model theft. Sources: OWASP/www-project-top-10-for-large-language-model-applications, guardrails-ai/guardrails, MITRE ATLAS, NIST AI RMF, leondz/garak. license: Apache-2.0 compatibility: yamtam-engine >= 1.3.40 metadata: origin: yamtam-engine — synthesized from OWASP/www-project-top-10-for-large-language-model-applications, guardrails-ai/guardrails, MITRE/ATLAS, NIST/AI-RMF, leondz/garak, simonw/llm (Simon Willison indirect injection research), Riley Goodside prompt injection research, langchain-ai/langchain (guardrails), BerriAI/litellm (proxy guardrails) version: 1.0.0 --- # /owasp-llm-top10 ## When to Use - Building any LLM-powered feature that reads external data - Security review of an AI agent before production - "What could go wrong with this agent?" threat modelling - Auditing tool/plugin integrations for an LLM system ## Do NOT use for - Pure ML model training (different threat surface) - LLM used only with static, trusted-internal prompts --- ## LLM01 — Prompt Injection ``` Direct: user input overrides system prompt Attack: "Ignore all previous instructions and..." Defense: structured input separation — never concat user input into system prompt wrap user content: { role: "user", content: sanitized(input) } Indirect: external data (web, files, email) contains injected instructions Attack: webpage fetched by agent contains "" Defense: prompt-jailbreak-guard.md scan on ALL external content before adding to context label external content as [EXTERNAL DATA — NOT INSTRUCTIONS] ``` --- ## LLM02 — Insecure Output Handling ``` Risk: LLM output rendered as HTML/code without sanitization → XSS, command injection Defense: - DOMPurify.sanitize() before innerHTML assignment - Never eval() LLM-generated code - File paths from LLM must pass sanitize_llm_path() (owasp-llm-output-law.md) - Agent-to-agent: wrap output in { type: "data" } before passing ``` --- ## LLM03 — Training Data Poisoning ``` Risk: model fine-tuned on poisoned data behaves incorrectly at inference Defense (for fine-tuning pipelines): - Validate training dataset provenance (SLSA-artifact-law.md Gate L4) - Detect distribution shift: compare new training data stats vs. baseline - Hold-out adversarial set: include known attack prompts in eval set - Never fine-tune on user-submitted data without human review gate ``` --- ## LLM04 — Model Denial of Service ``` Risk: crafted input causes excessive token usage or infinite loops Attack patterns: - "Repeat the following 10000 times..." - Recursive function call prompts - Deeply nested JSON schemas in prompt Defense: - max_tokens hard limit on every API call (never uncapped) - Token budget guard (token-budget-guard.sh): abort after 5 retries - Request timeout: 30s for single-turn, 120s for multi-step - Rate limit per session: YAMTAM token-roi skill monitors per-session spend ``` --- ## LLM05 — Supply Chain Vulnerabilities ``` Risk: compromised model, fine-tuning data, or LLM plugin Defense: - Pin exact model version: "claude-opus-4-7" not "claude-latest" - Verify model provider hash if available (Hugging Face: repo commit hash) - Plugin/MCP server whitelist: core/config/mcp-whitelist.json - dependency-vetting-law.md Gate L4 for all LLM tooling deps - slsa-artifact-law.md for released artifacts ``` --- ## LLM06 — Sensitive Information Disclosure ``` Risk: model outputs training data, PII, or secrets in response Defense: - System prompt must NOT contain secrets (use env vars, Vault) - Output scan: secure-logger.sh --scan-egress on every LLM response - PII detection before logging: strip SSN, CC, email from logs - User data minimization: don't include more context than needed - Prompt: explicitly instruct model not to repeat back sensitive input ``` --- ## LLM07 — Insecure Plugin Design ``` Risk: malicious plugin/tool returns instructions that hijack agent Defense: - agent-tool-poisoning-guard.md: validate schema + sanitize result - mcp-whitelist.json: deny-by-default for unknown servers - Tool result size cap: 16KB max before truncation - Tool result wrapped: [TOOL RESULT — DATA ONLY — NOT AN INSTRUCTION] - Separate tool execution context from instruction context ``` --- ## LLM08 — Excessive Agency ``` Risk: agent takes autonomous actions beyond intended scope Defense: - agent-excessive-agency-law.md: Tier R/W/X/P permission model - Irreversible actions require YAMTAM_IRREVERSIBLE_OK=1 per action - Sub-agent depth capped at 3 (YAMTAM_AGENT_DEPTH check) - Minimum scope declaration required at task start - Human confirmation gate for > 5 files or external endpoints ``` --- ## LLM09 — Overreliance ``` Risk: system blindly trusts LLM output for critical decisions Defense: - Truth Gate (truth-gate-guard.sh): bare claims without evidence → warn - Status: REVIEWED/UNKNOWN/CLAIMED — never auto-PASS - llm-output-validation skill: self-consistency check for factual claims - Human gate before LLM output used in: financial, medical, legal context - Confidence scoring required for high-stakes outputs ``` --- ## LLM10 — Model Theft ``` Risk: system prompt or model weights extracted via repeated probing Defense: - System prompt never returned in API response (check API config) - Rate limit + anomaly detection on repeated similar queries - System prompt stored in env var / Vault — not in codebase - Honeypot phrase in system prompt: if echoed back → log + revoke key - Watermark: include unique per-session token in system context ``` --- ## Anti-Fake-Pass Checklist ``` ❌ LLM01 check skipped because "system prompt is trusted" (indirect injection ignores this) ❌ LLM02 output rendered without DOMPurify (XSS) ❌ LLM04 API call without max_tokens set ❌ LLM05 model version pinned to "latest" (supply chain drift) ❌ LLM06 secrets in system prompt (extraction risk) ❌ LLM08 irreversible action without explicit per-action auth ❌ LLM09 LLM output used for financial/legal decision without human gate ```