--- name: validation-frameworks description: Problem and solution validation methodologies, assumption testing, and MVP validation experiments --- # Validation Frameworks Frameworks for validating problems, solutions, and product assumptions before committing to full development. ## When to Use This Skill **Auto-loaded by agents**: - `research-ops` - For problem/solution validation and assumption testing **Use when you need**: - Validating product ideas before building - Testing assumptions and hypotheses - De-risking product decisions - Running MVP experiments - Validating problem-solution fit - Testing willingness to pay - Evaluating technical feasibility ## Core Principle: Reduce Risk Through Learning Building the wrong thing is expensive. Validation reduces risk by answering critical questions before major investment: - **Problem validation**: Is this a real problem worth solving? - **Solution validation**: Does our solution actually solve the problem? - **Market validation**: Will people pay for this? - **Usability validation**: Can people use it? - **Technical validation**: Can we build it? ## The Validation Spectrum Validation isn't binary (validated vs. not validated). It's a spectrum of confidence: ``` Wild Guess → Hypothesis → Validated Hypothesis → High Confidence → Proven ``` Early stage: Cheap, fast tests (low confidence gain) Later stage: More expensive tests (high confidence gain) **Example progression**: 1. **Assumption**: "Busy parents struggle to plan healthy meals" 2. **Interview 5 parents** → Some validation (small sample) 3. **Survey 100 parents** → More validation (larger sample) 4. **Prototype test with 20 parents** → Strong validation (behavior observed) 5. **Launch MVP, track engagement** → Very strong validation (real usage) 6. **Measure retention after 3 months** → Proven (sustained behavior) ## Problem Validation vs. Solution Validation ### Problem Validation **Question**: "Is this a problem worth solving?" **Goal**: Confirm that: - The problem exists - It's painful enough that people want it solved - It's common enough to matter - Current solutions are inadequate **Methods**: - Customer interviews (discovery-focused) - Ethnographic observation - Surveys about pain points - Data analysis (support tickets, analytics) - Jobs-to-be-Done interviews **Red flags that stop here**: - No one cares about this problem - Existing solutions work fine - Problem only affects tiny niche - Pain point is mild annoyance, not real pain **Output**: Problem statement + evidence it's worth solving See `assets/problem-validation-canvas.md` for a ready-to-use framework. --- ### Solution Validation **Question**: "Does our solution solve the problem?" **Goal**: Confirm that: - Our solution addresses the problem - Users understand it - Users would use it - It's better than alternatives - Value proposition resonates **Methods**: - Prototype testing - Landing page tests - Concierge MVP (manual solution) - Wizard of Oz (fake backend) - Pre-sales or waitlist signups **Red flags**: - Users don't understand the solution - They prefer their current workaround - Would use it but not pay for it - Solves wrong part of the problem **Output**: Validated solution concept + evidence of demand See `assets/solution-validation-checklist.md` for validation steps. --- ## The Assumption-Validation Cycle ### 1. Identify Assumptions Every product idea rests on assumptions. Make them explicit. **Types of assumptions**: **Desirability**: Will people want this? - "Users want to track their habits" - "They'll pay $10/month for this" - "They'll invite their friends" **Feasibility**: Can we build this? - "We can process data in under 1 second" - "We can integrate with their existing tools" - "Our team can build this in 3 months" **Viability**: Should we build this? - "Customer acquisition cost will be under $50" - "Retention will be above 40% after 30 days" - "We can reach 10k users in 12 months" **Best practice**: Write assumptions as testable hypotheses - Not: "Users need this feature" - Yes: "At least 60% of users will use this feature weekly" --- ### 2. Prioritize Assumptions to Test Not all assumptions are equally important. Prioritize by: **Risk**: How uncertain are we? (Higher risk = higher priority) **Impact**: What happens if we're wrong? (Higher impact = higher priority) **Prioritization matrix**: | Risk/Impact | High Impact | Low Impact | |------------|-------------|------------| | **High Risk** | **Test first** | Test second | | **Low Risk** | Test second | Maybe skip | **Riskiest assumptions** (test these first): - Leap-of-faith assumptions the entire business depends on - Things you've never done before - Assumptions about user behavior with no data - Technical feasibility of core functionality **Lower-risk assumptions** (test later or assume): - Based on prior experience - Easy to change if wrong - Industry best practices - Minor features --- ### 3. Design Validation Experiments For each assumption, design the cheapest test that could prove it wrong. **Experiment design principles**: **1. Falsifiable**: Can produce evidence that assumption is wrong **2. Specific**: Clear success/failure criteria defined upfront **3. Minimal**: Smallest effort needed to learn **4. Fast**: Results in days/weeks, not months **5. Ethical**: Don't mislead or harm users **The Experiment Canvas**: See `assets/validation-experiment-template.md` --- ### 4. Run the Experiment **Before starting**: - [ ] Define success criteria ("At least 40% will click") - [ ] Set sample size ("Test with 50 users") - [ ] Decide timeframe ("Run for 1 week") - [ ] Identify what success/failure would mean for product **During**: - Track metrics rigorously - Document unexpected learnings - Don't change experiment mid-flight **After**: - Analyze results honestly (avoid confirmation bias) - Document what you learned - Decide: Pivot, persevere, or iterate --- ## Validation Methods by Fidelity ### Low-Fidelity (Hours to Days) **1. Customer Interviews** - **Cost**: Very low (just time) - **Validates**: Problem existence, pain level, current solutions - **Limitations**: What people say ≠ what they do - **Best for**: Early problem validation **2. Surveys** - **Cost**: Low - **Validates**: Problem prevalence, feature preferences - **Limitations**: Response bias, can't probe deeply - **Best for**: Quantifying what you learned qualitatively **3. Landing Page Test** - **Cost**: Low (1-2 days to build) - **Validates**: Interest in solution, value proposition clarity - **Measure**: Email signups, clicks to "Get Started" - **Best for**: Demand validation before building **4. Paper Prototypes** - **Cost**: Very low (sketch on paper/whiteboard) - **Validates**: Concept understanding, workflow feasibility - **Limitations**: Low realism - **Best for**: Very early solution concepts --- ### Medium-Fidelity (Days to Weeks) **5. Clickable Prototypes** - **Cost**: Medium (design tool, 2-5 days) - **Validates**: User flow, interaction patterns, comprehension - **Tools**: Figma, Sketch, Adobe XD - **Best for**: Usability validation pre-development **6. Concierge MVP** - **Cost**: Medium (your time delivering manually) - **Validates**: Value of outcome, willingness to use/pay - **Example**: Manually curate recommendations before building algorithm - **Best for**: Validating value before automation **7. Wizard of Oz MVP** - **Cost**: Medium (build facade, manual backend) - **Validates**: User behavior, feature usage, workflows - **Example**: "AI" feature that's actually humans behind the scenes - **Best for**: Testing expensive-to-build features --- ### High-Fidelity (Weeks to Months) **8. Functional Prototype** - **Cost**: High (weeks of development) - **Validates**: Technical feasibility, actual usage patterns - **Limitations**: Expensive if you're wrong - **Best for**: After other validation, final pre-launch check **9. Private Beta** - **Cost**: High (full build + support) - **Validates**: Real-world usage, retention, bugs - **Best for**: Pre-launch final validation with early adopters **10. Public MVP** - **Cost**: Very high (full product) - **Validates**: Product-market fit, business model viability - **Best for**: After all other validation passed --- ## Validation Metrics and Success Criteria ### Quantitative Validation Metrics **Interest/Demand**: - Landing page conversion rate - Waitlist signup rate - Pre-order conversion - Prototype test completion rate **Usage**: - Feature adoption rate - Task completion rate - Time on task - Frequency of use **Engagement**: - DAU/MAU (daily/monthly active users) - Session length - Return rate - Feature usage depth **Value Realization**: - Aha moment rate (reached key action) - Time to value - Retention curves - Referral rate **Business**: - Conversion to paid - Lifetime value (LTV) - Customer acquisition cost (CAC) - Churn rate --- ### Qualitative Validation Signals **Strong positive signals**: - Users ask when they can buy it - Users describe it to others excitedly - Users find use cases you didn't imagine - Users return without prompting - Users complain when you take it away **Weak positive signals**: - "That's interesting" - "I might use that" - Polite feedback with no follow-up - Theoretical interest but no commitment **Negative signals**: - Confusion about value proposition - Preference for existing solution - Feature requests that miss the point - Can't articulate who would use it - Ghosting after initial interest --- ## Setting Success Criteria **Before running experiment**, define what success looks like. **Framework**: Set three thresholds 1. **Strong success**: Clear green light, proceed confidently 2. **Moderate success**: Promising but needs iteration 3. **Failure**: Pivot or abandon **Example: Landing page test** - Strong success: > 30% email signup rate - Moderate success: 15-30% signup rate - Failure: < 15% signup rate **Example: Prototype test** - Strong success: 8/10 users complete task, would use weekly - Moderate success: 5-7/10 complete, would use monthly - Failure: < 5/10 complete or no usage intent **Important**: Decide criteria before seeing results to avoid bias. --- ## Teresa Torres Continuous Discovery Validation ### Opportunity Solution Trees Map opportunities (user needs) to solutions to validate: ``` Outcome └── Opportunity 1 ├── Solution A ├── Solution B └── Solution C └── Opportunity 2 └── Solution D ``` **Validate each level**: 1. Outcome: Is this the right goal? 2. Opportunities: Are these real user needs? 3. Solutions: Will this solution address the opportunity? ### Assumption Testing For each solution, map assumptions: **Desirability assumptions**: Will users want this? **Usability assumptions**: Can users use this? **Feasibility assumptions**: Can we build this? **Viability assumptions**: Should we build this? Then test riskiest assumptions first with smallest possible experiments. ### Weekly Touchpoints Continuous discovery = continuous validation: - Weekly customer interviews (problem + solution validation) - Weekly prototype tests with 2-3 users - Weekly assumption tests (small experiments) **Goal**: Continuous evidence flow, not one-time validation. See `references/lean-startup-validation.md` and `references/assumption-testing-methods.md` for detailed methodologies. --- ## Common Validation Anti-Patterns ### 1. Fake Validation **What it looks like**: - Asking friends and family (they'll lie to be nice) - Leading questions ("Wouldn't you love...?") - Testing with employees - Cherry-picking positive feedback **Fix**: Talk to real users, ask open-ended questions, seek disconfirming evidence. --- ### 2. Analysis Paralysis **What it looks like**: - Endless research without decisions - Testing everything before building anything - Demanding statistical significance with 3 data points **Fix**: Accept uncertainty, set decision deadlines, bias toward action. --- ### 3. Confirmation Bias **What it looks like**: - Only hearing what confirms existing beliefs - Dismissing negative feedback as "they don't get it" - Stopping research when you hear what you wanted **Fix**: Actively seek disconfirming evidence, set falsifiable criteria upfront. --- ### 4. Vanity Validation **What it looks like**: - "I got 500 email signups!" (but 0 conversions) - "People loved the demo!" (but won't use it) - "We got great feedback!" (but all feature requests, no usage) **Fix**: Focus on behavior over opinions, retention over acquisition. --- ### 5. Building Instead of Validating **What it looks like**: - "Let's build it and see if anyone uses it" - "It'll only take 2 weeks" (takes 2 months) - Full build before any user contact **Fix**: Always do cheapest possible test first, build only after validation. --- ## Validation Workflow: End-to-End Example **Idea**: Mobile app for tracking baby sleep patterns ### Phase 1: Problem Validation **Assumption**: "New parents struggle to understand their baby's sleep patterns" **Experiment 1: Customer Interviews** - Interview 10 new parents - Ask about sleep tracking current methods, pain points - Success criteria: 7/10 express frustration with current methods **Result**: 9/10 keep notes on paper, all frustrated by inconsistency **Conclusion**: Problem validated, proceed. --- ### Phase 2: Solution Validation (Concept) **Assumption**: "Parents would use an app that auto-tracks sleep from smart monitor" **Experiment 2: Landing Page** - Create page describing solution - Run ads targeting new parents - Success criteria: 20% email signup rate **Result**: 8% signup rate **Conclusion**: Interest is lower than hoped. Talk to non-signups to understand why. **Learning**: They don't have smart monitors, manual entry would be fine. --- ### Phase 3: Solution Validation (Refined) **Assumption**: "Parents will manually log sleep times if it's fast" **Experiment 3: Concierge MVP** - Ask 5 parents to text sleep times - Manually create charts/insights - Send back daily summaries - Success criteria: 4/5 use it for 2 weeks **Result**: All 5 use it for 2 weeks, ask to keep using **Conclusion**: Strong validation, build simple MVP. --- ### Phase 4: Build & Validate MVP **Build**: Simple app with manual time entry + basic charts **Experiment 4: Private Beta** - 30 parents use MVP for 4 weeks - Success criteria: 60% retention after 2 weeks, 40% after 4 weeks **Result**: 70% 2-week retention, 50% 4-week retention **Conclusion**: Exceeded criteria, prepare for launch. --- This workflow: problem → concept → refined solution → MVP took 8 weeks and cost minimal development before validation. --- ## Validation Checklist by Stage ### Idea Stage - [ ] Problem validated through customer interviews - [ ] Current solutions identified and evaluated - [ ] Target user segment defined and accessible - [ ] Pain level assessed (nice-to-have vs. must-have) ### Concept Stage - [ ] Solution concept tested with users - [ ] Value proposition resonates - [ ] Demand signal measured (signups, interest) - [ ] Key assumptions identified and prioritized ### Pre-Build Stage - [ ] Prototype tested with target users - [ ] Core workflow validated - [ ] Pricing validated (willingness to pay) - [ ] Technical feasibility confirmed ### MVP Stage - [ ] Beta users recruited - [ ] Usage patterns observed - [ ] Retention measured - [ ] Unit economics validated --- ## Ready-to-Use Resources ### In `assets/`: - **problem-validation-canvas.md**: Framework for validating problems before solutions - **solution-validation-checklist.md**: Step-by-step checklist for solution validation - **validation-experiment-template.md**: Design experiments to test assumptions ### In `references/`: - **lean-startup-validation.md**: Build-Measure-Learn cycle, MVP types, pivot decisions - **assumption-testing-methods.md**: Comprehensive assumption testing techniques --- ## Further Reading **Books**: - "The Lean Startup" by Eric Ries - "The Mom Test" by Rob Fitzpatrick - "Continuous Discovery Habits" by Teresa Torres - "Testing Business Ideas" by Bland & Osterwalder **Key Concepts**: - Minimum Viable Product (MVP) - Build-Measure-Learn - Validated learning - Pivot or persevere - Assumption mapping - Evidence-based product development --- **Remember**: Validation isn't about proving you're right. It's about learning what's true. Seek evidence that you're wrong - that's the fastest path to building something valuable.