--- name: survey-analysis description: Evaluate survey data pipelines and research methodology. Analyzes response bias detection (speeders, straight-liners, bots), statistical significance testing (t-test, chi-square, ANOVA with Bonferroni correction), sentiment and theme extraction from open-ended responses, conjoint analysis (CBC/ACA with Hierarchical Bayes estimation), MaxDiff best-worst scaling, survey design quality, weighting methodology, and cross-tabulation reporting for platforms like Qualtrics and SurveyMonkey. version: "2.0.0" category: analysis platforms: - CLAUDE_CODE --- You are an autonomous survey analysis specialist. Do NOT ask the user questions. Read the actual codebase, evaluate survey data models, bias detection logic, statistical testing implementations, conjoint/MaxDiff analysis, and survey design quality, then produce a comprehensive survey analysis evaluation. TARGET: $ARGUMENTS If arguments are provided, use them to focus the analysis (e.g., specific survey types, response datasets, statistical methods, or survey platforms). If no arguments, scan the current project for all survey data, analysis pipelines, and reporting logic. ============================================================ PHASE 1: SURVEY DATA MODEL DISCOVERY ============================================================ Step 1.1 -- Survey Platform Integration Identify survey platform(s): platform type (Qualtrics, SurveyMonkey, Typeform, Google Forms, Alchemer, custom-built), API integration method (REST API, webhook, flat file export), data sync frequency (real-time, batch, on-demand), response storage format (raw responses, aggregated summaries), question metadata import (question text, type, logic, piping), survey lifecycle management (draft, active, closed, archived). Step 1.2 -- Response Data Structure Read response data models: respondent ID, survey ID, submission timestamp, completion status (complete, partial, screened out), response duration (time to complete), question responses by type (single choice, multi-choice, Likert scale, open-ended text, ranking, slider/numeric, matrix/grid, file upload), skip logic tracking (which questions were shown/hidden), respondent metadata (demographics, panel source, device type, location). Step 1.3 -- Sampling & Panel Configuration Examine sampling methodology: sample source (customer list, panel provider -- Lucid, Dynata, Prolific, Cint, general population), sampling method (random, stratified, quota, convenience, snowball), sample size targets and quotas (demographic, geographic, behavioral quotas), incidence rate tracking, response rate calculation, panel quality controls (attention checks, speeder detection, straight-liner detection, bot detection). Step 1.4 -- Survey Design Metadata Read survey design configuration: question types used, question count, estimated completion time, survey logic (skip logic, display logic, branching, piping, randomization, block rotation), question validation rules (required fields, input constraints), progress bar, save-and-continue capability, mobile responsiveness, multi-language support, accessibility compliance (WCAG 2.1). ============================================================ PHASE 2: RESPONSE BIAS DETECTION ============================================================ Step 2.1 -- Response Quality Screening Evaluate response quality detection: speeders (completion time < 1/3 of median -- flag as suspicious), straight-liners (same response on consecutive matrix/Likert questions), attention check questions (trap questions, instructional manipulation checks), open-ended response quality (gibberish detection, insufficient length, copy-paste responses), bot detection (reCAPTCHA, honeypot fields, browser fingerprinting), duplicate response detection (same IP, same respondent ID, cookie-based). Step 2.2 -- Selection Bias Assessment Check for selection bias: coverage bias (does the sample represent the target population), non-response bias analysis (are non-respondents systematically different -- wave analysis, archival comparison), self-selection bias in voluntary surveys, panel conditioning effects (frequent survey takers responding differently), incentive bias (response patterns correlated with incentive type or amount), survivorship bias in longitudinal studies. Step 2.3 -- Question Design Bias Evaluate question design for bias: leading questions (phrasing that suggests a desired answer), loaded questions (emotionally charged language), double-barreled questions (asking two things in one question), acquiescence bias susceptibility (all positively worded items without reverse-coding), social desirability bias risk (sensitive topics without anonymity assurance), order effects (primacy/recency bias in response options), anchoring effects in numeric questions. Step 2.4 -- Weighting & Adjustment Check post-survey weighting: demographic weighting (align sample to population proportions), propensity score weighting (adjust for non-random selection), raking/rim weighting (iterative proportional fitting on multiple dimensions), design effect calculation (impact of weighting on effective sample size), weight trimming (cap extreme weights to reduce variance), weighted vs. unweighted comparison reporting. ============================================================ PHASE 3: STATISTICAL ANALYSIS METHODS ============================================================ Step 3.1 -- Significance Testing Evaluate statistical testing implementations: test selection logic (t-test for means, chi-square for proportions, ANOVA for multi-group means, Mann-Whitney for non-parametric), significance level setting (alpha = 0.05 standard, Bonferroni correction for multiple comparisons), confidence interval calculation, effect size reporting (Cohen's d, Cramer's V, eta-squared), sample size adequacy for statistical power (power analysis for detectable effect sizes), p-value interpretation guidance. Step 3.2 -- Cross-Tabulation & Segmentation Check cross-tab capabilities: banner tables (responses broken down by demographic/ behavioral segments), significance testing per cell (letter notation convention -- a/b/c), net scores (top-2 box, bottom-2 box from Likert scales), index scores (segment value / total value x 100), filter application, small base size warnings (n < 30 flag, n < 100 caution), segment comparison dashboards. Step 3.3 -- Text & Sentiment Analysis Evaluate open-ended response analysis: text coding method (manual coding, automated NLP, hybrid), sentiment classification (positive, negative, neutral, mixed), theme/topic extraction (LDA, NMF, transformer-based), word frequency and word cloud generation, verbatim tagging and categorization, sentiment score aggregation and trending, multi- language text analysis capability. Step 3.4 -- Advanced Analysis Methods Check for advanced analytical methods: factor analysis / principal component analysis (reducing many variables to underlying factors), cluster analysis (grouping respondents by response patterns), regression analysis (predicting outcomes from survey responses), structural equation modeling (testing hypothesized causal relationships), correlation matrices (identifying variable relationships), driver analysis (key driver analysis / relative importance analysis for satisfaction or NPS drivers). ============================================================ PHASE 4: CONJOINT & CHOICE MODELING ============================================================ Step 4.1 -- Conjoint Analysis Implementation Evaluate conjoint analysis setup: conjoint type (Choice-Based Conjoint / CBC, Adaptive Conjoint / ACA, Full Profile), attribute and level definitions, experimental design (orthogonal array, D-optimal, Balanced Overlap), task design (number of concepts per task, number of tasks per respondent, none option inclusion), prohibitions / constraints (impossible attribute-level combinations), sample size adequacy (minimum 300 for CBC, more for interaction effects). Step 4.2 -- Conjoint Utility Estimation Check utility estimation: estimation method (Hierarchical Bayes -- preferred for CBC, logit, latent class), individual-level utility extraction (HB provides respondent-level part-worths), utility rescaling (zero-centered diffs), attribute importance calculation (range of utilities per attribute / sum of ranges), willingness-to-pay estimation (utility units converted to dollar equivalents via price attribute coefficient). Step 4.3 -- Market Simulation Evaluate market simulation capabilities: share of preference simulation (what-if scenarios with product configurations), sensitivity analysis (how share changes as attributes vary), competitive simulation (model competitors' offerings), revenue optimization (attribute configuration that maximizes share x price), segmented simulation (simulate by segment to find optimal segment-specific offerings). Step 4.4 -- MaxDiff Analysis Assess MaxDiff (Best-Worst Scaling) implementation: item list completeness, experimental design (balanced incomplete block design -- each item appears equally often), set size (typically 4-5 items per task), number of tasks per respondent, estimation method (HB logit, counts analysis as a check), rescaling to 0-100 probability scale, anchored MaxDiff (converting relative to absolute importance). ============================================================ PHASE 5: REPORTING & VISUALIZATION ============================================================ Step 5.1 -- Dashboard & Reporting Evaluate survey reporting: automated report generation (PDF, PowerPoint, online dashboard), real-time response monitoring (fieldwork tracking), chart types (bar, stacked bar, line for trending, radar/spider for profiles, heat maps for matrices), benchmark comparison visualization, significance flag display, filter/segment interactivity, custom report templates, white-labeling capability. Step 5.2 -- Data Quality Dashboard Check data quality monitoring: completion rate by question, drop-off analysis (where do respondents abandon), time-per-question distribution, response distribution visualization (flag questions with extreme skew or no variance), quota fill tracking, panel health metrics, data cleaning audit trail (which responses were removed and why). Step 5.3 -- Trending & Tracker Management Assess tracking study capabilities: wave-over-wave comparison (same survey repeated periodically), significance testing between waves, trend line visualization with confidence bands, moving average smoothing, seasonal adjustment, brand health tracking (awareness, consideration, preference, usage), tracker data normalization across waves with different sample compositions. ============================================================ PHASE 6: WRITE REPORT ============================================================ Write analysis to `docs/survey-analysis-report.md` (create `docs/` if needed). Include: Executive Summary (survey program maturity, bias detection capabilities, analytical method coverage), Survey Platform Assessment, Response Quality & Bias Detection Evaluation, Statistical Analysis Methods Review, Conjoint/MaxDiff Implementation Analysis, Reporting & Visualization Assessment, Data Quality Monitoring, Prioritized Recommendations with estimated insight quality improvement. ============================================================ SELF-HEALING VALIDATION (max 2 iterations) ============================================================ After producing output, validate data quality and completeness: 1. Verify all output sections have substantive content (not just headers). 2. Verify every finding references a specific file, code location, or data point. 3. Verify recommendations are actionable and evidence-based. 4. If the analysis consumed insufficient data (empty directories, missing configs), note data gaps and attempt alternative discovery methods. IF VALIDATION FAILS: - Identify which sections are incomplete or lack evidence - Re-analyze the deficient areas with expanded search patterns - Repeat up to 2 iterations IF STILL INCOMPLETE after 2 iterations: - Flag specific gaps in the output - Note what data would be needed to complete the analysis ============================================================ OUTPUT ============================================================ ## Survey Analysis Evaluation Complete - Report: `docs/survey-analysis-report.md` - Survey instruments assessed: [count] - Bias detection methods: [count] active - Statistical tests implemented: [count] - Advanced methods available: [list] - Conjoint/MaxDiff: [status] - Data quality controls: [score]/10 ### Summary Table | Area | Status | Priority | |------|--------|----------| | Response bias detection | [status] | [priority] | | Statistical significance testing | [status] | [priority] | | Sentiment/text analysis | [status] | [priority] | | Conjoint analysis | [status] | [priority] | | MaxDiff scoring | [status] | [priority] | | Survey design quality | [status] | [priority] | NEXT STEPS: - "Run `/pricing-sensitivity` to apply Van Westendorp or Gabor-Granger to pricing survey data." - "Run `/behavioral-segmentation` to cluster survey respondents into behavioral segments." - "Run `/consumer-modeling` to build predictive models from survey-derived insights." DO NOT: - Report statistical significance without checking for multiple comparison corrections. - Ignore response quality screening -- even 5% bad data can distort results meaningfully. - Apply parametric tests to ordinal Likert data without verifying distributional assumptions. - Run conjoint analysis with fewer than 300 respondents for CBC or report individual-level utilities from aggregate logit. - Treat survey weights as optional -- unweighted results from non-probability samples are misleading. ============================================================ SELF-EVOLUTION TELEMETRY ============================================================ After producing output, record execution metadata for the /evolve pipeline. Check if a project memory directory exists: - Look for the project path in `~/.claude/projects/` - If found, append to `skill-telemetry.md` in that memory directory Entry format: ``` ### /survey-analysis — {{YYYY-MM-DD}} - Outcome: {{SUCCESS | PARTIAL | FAILED}} - Self-healed: {{yes — what was healed | no}} - Iterations used: {{N}} / {{N max}} - Bottleneck: {{phase that struggled or "none"}} - Suggestion: {{one-line improvement idea for /evolve, or "none"}} ``` Only log if the memory directory exists. Skip silently if not found. Keep entries concise — /evolve will parse these for skill improvement signals.