---
name: goblin-recon
description: Operate the Goblin Recon agent — GenX Academy's semi-autonomous content intelligence system. Uses a professional router/workflow architecture with Social Pulse, Clip Mine, and Clip Vault. Preserves the 3-layer clip chain (Trend Radar → Source Hunter → Moment Finder) for finding trending AI stories, locating source videos, and extracting 15-60 second clip moments. Covers scan modes, social extraction reliability, script usage, brand gate application, profile setup, and end-to-end testing workflow.
category: genx-marketing
---

# Goblin Recon — Operational Skill

## What It Is

Goblin Recon is the intelligence division of the Goblin Bureau (GenX's AI agent suite). It is a semi-autonomous content intelligence system with this professional operating pattern:

```text
Router -> Workflow -> Tools -> Normalized Data -> Score -> Human Gate -> Memory
```

The agent does not scan every platform or invoke every tool by default. It routes the user's request into one primary workflow, uses the minimum reliable tools, presents decision-ready output, and stores useful memory.

For a plain-language command list, use `COMMANDS.md` as the team-facing reference.

When a user asks for progress or repeats a question, do not paste the same answer again. Start with what changed since the last answer, then state what is still true and the smallest useful next action.

## Workflows

Goblin Recon has three core workflows:

```mermaid
flowchart TD
    A[User Command] --> B[Intent Router]
    B --> C[Social Pulse]
    B --> D[Clip Mine]
    B --> E[Clip Vault]
    C --> C1[Rank trend/content opportunities]
    D --> D1[Create editor-ready clip briefs]
    E --> E1[Retrieve clips and update statuses]
```

### Workflow 1: Social Pulse
**Purpose:** Content ideas, blogs, carousels, content strategy inspiration.
**Sources:** Instagram → TikTok → X → Reddit → Tech News
**Output:** Trending topics, hook styles, reel formats, carousel angles, blog ideas.
**NOT for:** Direct video clips.

Commands:
| Command | What It Does |
|---------|--------------|
| `run fast scan` | Low-stress daily scan using reliable sources first |
| `run deep social scan` | Deeper IG/TikTok-first social scan with fallback when blocked |
| `run signal scan` | First-mover scan across public early-signal sources |
| `manual scan this [URL/screenshot/caption]` | Normalize and score human-provided social material |
| `run social pulse` | Full scan across IG/TikTok/X/Reddit/News for AI trends |
| `what's trending on Instagram` | IG-only creator account scan |
| `what's trending on TikTok` | TikTok-only trend scan |
| `blog ideas` | Social Pulse filtered for long-form content angles |
| `carousel ideas` | Social Pulse filtered for carousel-worthy topics |
| `content strategy this week` | Social Pulse + editorial calendar suggestions |

### Workflow 2: Clip Mine
**Purpose:** Direct video clips for the faceless Instagram page.
**Sources:** YouTube podcasts → Interviews → Keynotes
**Output:** Timestamped clips (15-60s), transcript quotes, engagement analytics, editor-ready briefs.
**This goes straight to editors.** They download the clip and produce the reel.

Commands:
| Command | What It Does |
|---------|--------------|
| `run clip mine` | Find best podcast clips from trending AI stories |
| `find clips about [topic]` | Source Hunter + Moment Finder for specific topic |
| `find the moment in [URL]` | Extract best clip from a specific video |

Clip Mine preserves the original core chain:

```text
Trend Radar -> Source Hunter -> Moment Finder -> Brand Gate -> Human Gate
```

### Workflow 3: Clip Vault
**Purpose:** Persistent memory for approved, shelved, and production-status clips.
**Storage:** `vault/clips.db`, `vault/briefs/`, `memory/trend-history.md`.
**Output:** Ready clips, duplicate warnings, regenerated briefs, status updates.

Commands:
| Command | What It Does |
|---------|--------------|
| `what clips are ready` | Show approved clips awaiting editor handoff |
| `search clips about [topic]` | Search stored clips by topic/source/summary/caption |
| `show clip [clip_id]` | Show one stored clip record |
| `update clip status` | Move a clip through approved/in_production/scheduled/posted/shelved |

### Categorization (Core Workflows)
Every item — Social Pulse idea or Clip Mine clip — is tagged by type:

| Category | What It Means |
|----------|--------------|
| **Latest AI News** | Breaking developments, product launches, policy changes |
| **Controversial/Polarizing** | Debates, backlash, hot takes, culture-war adjacent |
| **Upgrade/Democratization** | "Anyone can now do X," tool tutorials, barrier collapsing |
| **Analytical/Deep-dive** | Strategic insights, economic analysis, future predictions |

### General Commands
| Command | What It Does |
|---------|--------------|
| `run full scan` | Social Pulse + Clip Mine in sequence |
| `what formats are working?` | Current winning reel formats from IG/TikTok |
| `run competitor scan` | Competitor Scout |

## Intent Router

Before using tools, classify the user's request:

| User Intent | Primary Workflow | Notes |
|-------------|------------------|-------|
| Find trends, ideas, hooks, or formats | Social Pulse | Ask or infer scan mode. |
| Find video sources or timestamped moments | Clip Mine | Run vault dedup before Moment Finder. |
| Retrieve approved/shelved clips | Clip Vault | Query `vault/clips.db` first. |
| Analyze competitors | Competitor Scout | Keep separate from Social Pulse and Clip Mine. |
| Validate copy/content against brand | Brand Gate | Can run standalone. |
| Generate or validate outbound email hooks | Email Hook | Ask direction first, then run email gate. |

If a request mixes workflows, use the smallest useful sequence. Example: `run full scan` means Social Pulse first, then Clip Mine only for the 2-3 strongest candidates.

### Output Direction Pre-Check

Before creating brand-facing output, ask these questions in one short message:

1. Who is this for? B2C, B2B, or Both?
2. Where does it go? Faceless Instagram, personal brand, client work, internal use, email/outbound, or other?
3. What tone should it carry? Professional, casual, edgy, warm, wry, reflective, analytical/data-driven, bold, or platform-native?

Use the answer to set brand angle, destination, tone, format, scoring lens, and copy guardrails for the session. If the user skips the questions, default to Both / Faceless Instagram / professional and state that default before generating.

Skip this pre-check only for Clip Vault retrieval, Manual Assisted Scan where the user already supplied direction, and standalone brand checks on user-provided copy.

## Scan Modes

### Fast Scan
Use for daily low-stress discovery. Prefer reliable sources: YouTube, Reddit, Tech News, Product Hunt, and X/Twitter only when public access or approved API access is available. Do not depend on Instagram/TikTok extraction unless explicitly requested.

### Deep Social Scan
Use for weekly social-native discovery or important launches. Start with Instagram and TikTok public surfaces, then validate with X/Twitter, Reddit, and Tech News. If a platform blocks access, mark it blocked and move on.

### Manual Assisted Scan
Use when the human provides URLs, screenshots, captions, creator handles, or notes. Normalize the material into the social record schema, score it, and recommend whether it belongs in Social Pulse or Clip Mine.

### Signal Scan
Use for first-mover discovery when mainstream news is too slow. Scan public early-signal surfaces in this order: X/Twitter when approved/public, Hacker News, GitHub Trending, ArXiv, then Reddit only if public access works. Time gate: last 6 hours. If nothing clears the velocity threshold, return "nothing worth posting right now" instead of forcing weak ideas.

## Social Extraction Reliability Ladder

Use this order when social data is needed:

```text
Approved API or reliable public feed -> Public browser extraction -> Manual assisted input
```

Every social signal should be normalized before scoring:

```text
platform:
creator:
url:
published_date:
views:
likes:
comments:
caption:
hook:
format_type:
topic:
category:
why_it_is_trending:
can_genx_adapt_this:
confidence:
access_status:
```

Rules:
- Never bypass login, paywall, captcha, robots.txt, rate limits, or platform restrictions.
- Never use personal employee accounts for automation.
- If public extraction fails, set `access_status: blocked` and switch to manual assisted input only if the missing data is essential.
- Instagram and TikTok browser extraction are useful but fragile; do not build the whole workflow around them.

All social observations must pass through `goblin_recon.tools.social_intake` before scoring. This creates a stable intake layer for approved API data, public browser observations, and manual assisted inputs.

Examples:
```bash
.venv/bin/python -m goblin_recon.tools.social_intake --input vault/intake/social-signal.json
.venv/bin/python -m goblin_recon.tools.social_intake --url "https://www.instagram.com/reel/..." --topic "AI agents" --caption "..."
.venv/bin/python -m goblin_recon.tools.social_intake --input vault/intake/social-signal.json --store
```

Default store: `vault/social-signals.jsonl` (ignored by Git).

## Trend Detection Priority (CRITICAL)

For full Social Pulse and Deep Social Scan, **Instagram and TikTok first.** These platforms show what's ACTUALLY engaging — not just what journalists think is important. News sites (TechCrunch, Verge, VentureBeat, Ars Technica) are for **validation** — URLs, dates, journalistic verification. They are NOT the primary trend signal in social-native scans.

Fast Scan is the exception: it intentionally uses reliable sources first and may skip Instagram/TikTok unless explicitly requested.

Default social-native priority: **1. Instagram → 2. TikTok → 3. X/Twitter → 4. Reddit → 5. Tech News → 6. Product Hunt**

Instagram creator accounts to scan:
- @therundownai (491K) — carousel news digest
- @rowancheung (418K) — interview clips
- @inflecta.ai — narrative storytelling
- @ankitgupta.ai — AI tool showcases

Extract from Instagram: story, hook style, format type, view count, engagement metrics.
IG rules: public profiles only, no login bypass, stop if blocked. Min 50K views for signal.

## Project Location

The Goblin Recon project lives wherever you clone it. The structure:
```
goblin-recon/
├── SOUL.md          ← your identity file (copy to profile)
├── ARCHITECTURE.md  ← router, workflows, scan modes, and tool policy
├── AGENTS.md        ← agent constitution
├── config/          ← sources, scoring, brand-voice, security
├── memory/          ← brand-rules, trend/competitor history
├── goblin_recon/tools/ ← importable tool modules
├── scripts/         ← standalone setup, secret scan, and query helpers
├── templates/       ← output templates
├── personal-dumpground/ ← local-only notes, ignored by Git
└── mcp.json         ← MCP server config (all optional)
```

Key files:
- `SOUL.md` — pre-made identity file. Copy to `~/.hermes/profiles/goblin-recon/SOUL.md`
- `ARCHITECTURE.md` — professional system map: router, workflows, scan modes, social extraction ladder, tool policy, memory policy
- `AGENTS.md` — the agent's constitution (personality, rules, scoring, output format, trend priority)
- `personal-dumpground/SESSION_LOG.md` — optional local-only session notes, not shipped with the company repo
- `config/sources.yaml` — source priority: Instagram → TikTok → X → Reddit → News
- `config/content-sources.yaml` — YouTube channels, IG accounts, TikTok creators
- `config/scoring.yaml` — scoring dimensions (social_velocity, scroll_stop, etc.)
- `config/brand-voice.yaml` — brand voice rules, blacklist, nuance words
- `config/security.yaml` — data collection policies, API key rules, rate limits
- `memory/brand-rules.md` — operational brand memory for the agent
- `goblin_recon/tools/` — importable tool modules (transcripts, clips, scoring, brand gate, social intake)
- `scripts/` — standalone setup, secret scan, and query helpers
- `goblin_recon.tools.brand_gate` — pre-flight blacklist and nuance-word check for GenX-written copy
- `templates/` — output templates (social-pulse-report, clip-mine-brief, competitor-report; trend-report/content-brief are deprecated references)
- `mcp.json` — MCP server configuration (all optional, see pitfalls)

## Profile Setup (Hermes Desktop)

When a new user creates the goblin-recon profile, use the project setup script as the source of truth:

```bash
cd goblin-recon
bash scripts/setup.sh
```

The script installs the profile, SOUL.md, bundled skills, profile defaults, Python virtual environment, and dependencies.

### 1. SOUL.md Copy

A pre-made SOUL.md lives at the project root. From the project directory:

If setup fails to copy SOUL.md, copy it manually from the project root:
```bash
cp SOUL.md ~/.hermes/profiles/goblin-recon/SOUL.md
```

The SOUL.md contains everything the agent needs:
- Core identity and GenX Academy context (who we are, two brand doors, mission spine)
- Brand voice DNA (B2C/B2B tone, blacklist summary, nuance words)
- Personality and communication rules
- Trend detection philosophy (IG-first priority)
- Output standards (Decision-first, platform variants, phone-scannable)
- Security and compliance rules (compressed)
- Setup instructions for new users
- Maintenance guide (what to edit when things change)

See the file itself for the full content. It's self-documenting.

### 2. Skill Auto-Load
```bash
hermes config set skills.auto_load goblin-recon -p goblin-recon
```
This ensures the goblin-recon operational skill loads every time the profile starts. The agent always knows what it is.

### 3. Cherry-Picked Skills
Copy only the 5 marketing skills Goblin Recon needs (not all 55):
```bash
# From the source profile's desktop skills:
# competitor-profiling, social-content, copywriting, content-strategy, marketing-psychology
cp -r $SOURCE/skills/desktop/<skill> $GOBLIN/skills/desktop/<skill>
```
These cover: competitor research, social platform formats, caption writing, content planning, and engagement psychology. Skip: ad creative, email sequences, pricing, SEO, CRO, launch strategy, etc.

### 4. Model Config

Use whichever provider and model the company has approved. Example:

```bash
hermes -p goblin-recon config set model.provider openai
hermes -p goblin-recon config set model.default gpt-4o
hermes config set agent.max_turns 90 -p goblin-recon
hermes config set terminal.timeout 300 -p goblin-recon
```

Never paste or commit API keys. Store provider keys through Hermes secrets or another approved local secret method.

## Delegate Task Policy

NEVER use delegate_task/subagents for Fast Scan, Deep Social Scan, Signal Scan, single-source lookups, brand gate checks, or transcript extraction. Subagents do not reliably inherit Goblin Recon context and can waste tokens by brute-forcing browser navigation.

ONLY use delegate_task after data is already collected, and only for post-processing such as scoring, cross-referencing, report formatting, or counter-review. If you delegate, pass source URLs, query limits, blocked-source rules, brand rules, and expected output fields explicitly.

## How to Run Workflows

### Option A: Manual Sequential (Default)

Route first, then run only the needed workflow with direct tool use. For full Clip Mine, collect Layer 1 and Layer 2 data yourself, then run Layer 3 sequentially:

```
# Layer 1 (Trend Radar): direct browser/web/API extraction
# PRIORITY: Scan Instagram creator accounts FIRST (@therundownai, @rowancheung)
# Then TikTok hashtags, then X/Twitter, then tech news
# Return top 5-8 stories with: headline, IG views/engagement, format type, hook style
# Include both trending STORIES and trending FORMATS

# Layer 2 (Source Hunter): direct YouTube/social search
# Search YouTube, Instagram, TikTok for videos about top stories
# Return video title, channel, URL, publish date, duration, views, format, captions?

# Layer 3 (Moment Finder): Run sequentially after picking best source
# 1. Use get_youtube_transcript.py to pull transcript
# 2. Check vault/clips.db for duplicate source/time windows
# 3. Analyze transcript for best 15-60s moment (prioritize scroll_stop, quotability)
# 4. Validate with extract_clip.py
```

### Option B: Limited Delegation (Post-Processing Only)

Use delegation only after source data is already in hand. Suitable tasks: scoring candidate rows, cross-checking source dates, formatting a report, or counter-reviewing a recommendation.

## Script Usage

All scripts are in the project's `.venv`. Run from the project root:

```bash
.venv/bin/python -m goblin_recon.tools.youtube_tool "<video_id_or_url>"
```

### get_youtube_transcript.py
Extracts transcripts with timestamps. Output: JSON array of `{time, duration, text}`.
- Pass video ID or full YouTube URL
- `--languages en,zh-Hans` to try multiple languages (English preferred for GenX)
- Returns `{"error": "...", "recoverable": true}` on failure

### extract_clip.py
Validates clip metadata. Output: JSON with `url_with_timestamp`, `embed_url`, `duration`, `start_time`, `end_time`.
- Usage: `.venv/bin/python -m goblin_recon.tools.extract_clip <video_url> <start_sec> <end_sec>`
- Enforces 15–60 second duration
- Automatically generates YouTube timestamp links

### score_engagement.py
Calculates engagement velocity score (0–20). Output: JSON with `score`, `velocity_per_hour`, `hours_since_post`.
- Usage: `.venv/bin/python -m goblin_recon.tools.score_engagement <platform> <post_url_or_id> <ISO_timestamp> <views>`
- Platforms: twitter, reddit, youtube, instagram
- Platform-specific benchmarks for viral thresholds

### social_intake.py
Normalizes social media observations before Trend Radar scoring.
- Use for approved API records, public browser findings, and manual assisted input
- Infer platform from URL when possible
- Validate required fields: platform, URL, topic, access status
- Store local social signals: `.venv/bin/python -m goblin_recon.tools.social_intake --input vault/intake/social-signal.json --store`
- Do not store secrets, cookies, login-only data, or private personal data

### clip_store.py
Stores Clip Mine candidates in `vault/clips.db` for cross-session lookup, full-text search, and duplicate checks.
- Run with no args to initialize the database: `.venv/bin/python -m goblin_recon.tools.clip_store`
- Use from Hermes tool calls or local scripts to save approved or shelved clips with source URL, timestamps, status, scores, and summary fields
- Correct API: `save_clip({"source_url": "...", "start_sec": 10, "end_sec": 40, ...})`
- Compatibility API: `save_clip_kwargs(source_url="...", start_sec=10, end_sec=40, ...)`
- Count stored records with `get_clip_count()`
- Do not store full raw transcripts, API keys, cookies, or login-only source data

### query_clips.py
Retrieves stored clips without manually opening SQLite.
- Initialize: `.venv/bin/python scripts/query_clips.py init`
- List approved clips: `.venv/bin/python scripts/query_clips.py list --status approved`
- Search by topic/source/summary/caption: `.venv/bin/python scripts/query_clips.py list --query "AI agents"`
- Show one record: `.venv/bin/python scripts/query_clips.py show [clip_id]`
- Update workflow status: `.venv/bin/python scripts/query_clips.py update-status [clip_id] in_production --decision "editor picked it"`
- Export a markdown brief: `.venv/bin/python scripts/query_clips.py brief [clip_id] --output vault/briefs/[clip_id].md`

### check_secrets.py
Pre-commit security scan. Run before sharing or pushing:
```bash
.venv/bin/python scripts/check_secrets.py
```

### check_brand.py
Pre-flight brand gate helper for generated captions, summaries, hooks, and outbound copy:
```bash
.venv/bin/python -m goblin_recon.tools.brand_gate --text "Your caption or hook here"
.venv/bin/python -m goblin_recon.tools.brand_gate --file path/to/copy.md --json
```
A fail means rewrite or shelve before Human Gate.

### caption-tone Skill
Use `skills/caption-tone/SKILL.md` for caption and description tasks after Output Direction is clear. Default to professional GenX Academy copy, then ask whether the user wants another voice when the content would benefit from a casual, edgy, warm, wry, curious, reflective, analytical/data-driven, bold, or platform-native version. Run the brand gate on generated outward copy when feasible.

### email-hook Skill
Use `skills/email-hook/SKILL.md` for outbound email subject lines, openers, and short email drafts. Ask Output Direction first, select the campaign type from `config/email-campaigns.yaml`, then run `.venv/bin/python -m goblin_recon.tools.email_gate` before delivering final email copy.

## Clip Mine Scoring Criteria (7 Dimensions)

Every clip is scored out of 100 points. The agent applies these when scanning transcripts:

### 1. Scroll-Stop (15 pts) — THE MAIN TEST
"Would someone stop scrolling for this?"
- ✅ Concrete numbers, confrontational claims, revelations, emotional reactions
- ❌ Generic observations, pleasant conversation, "interesting" facts

### 2. Quotability (20 pts)
"Would someone screenshot this and share it?"
- ✅ Standalone soundbites, punchy phrasing, memorable analogies
- ❌ Rambling, needs context, requires knowing the speaker

### 3. Emotion (15 pts)
"Does it trigger a feeling?"
- ✅ Fear, awe, anger, excitement
- ❌ Flat delivery, neutral info, no stakes

### 4. Clarity (15 pts)
"Is the point clear without context?"
- ✅ First sentence tells the argument, no jargon, anyone understands
- ❌ Technical jargon, needs 20 minutes of prior context

### 5. Controversy (10 pts)
"Is this going to get comments?"
- ✅ Challenges conventional wisdom, contrarian strategy, polarizing take
- ❌ Safe takes, echoes the consensus

### 6. Visual Potential (10 pts)
"Can this work as a faceless reel with just text overlay?"
- ✅ Strong quote on screen, works with waveform + text
- ❌ Needs person's face to land, relies on visual demo

### 7. Brand Alignment (15 pts)
"Does this fit GenX Academy's voice?"
- ✅ B2B: Results not advice, operational detail, "here's what happened"
- ✅ B2C: Real science + real soul, truly seen, depth + play
- ❌ Hype language, hustle-bro, guru certainty, generic motivation

### Score Thresholds
| Score | Verdict |
|-------|---------|
| 85+ | Killer clip. Send to Human Gate. |
| 70-84 | Strong. Worth producing. |
| 60-69 | Decent. Produce only if nothing better passes. |
| Below 60 | Skip. Won't land. |

### What the Agent Hunts For
When scanning a transcript, these patterns win:

| Pattern | Example | Category |
|---------|---------|----------|
| Number reveals | "80% of code is now AI-authored" | Latest AI News |
| Strategy contradictions | "Being first is expensive. Being right matters." | Analytical |
| Barrier collapsing | "Two nights with Claude docs, made $238K" | Upgrade |
| Industry callouts | "Companies are poisoning their own AI answers" | Controversial |
| Future predictions | "Today's AI will look like flip phones in 3 years" | Analytical |
| Behind-the-scenes | "The real reason Microsoft walked away" | Controversial |

## Clip Mine: End-to-End Process

### Phase 1: AI Discovers (Goblin Recon)
```
User says: "run clip mine"
         ↓
Goblin Recon:
  ├─ Scans Instagram/TikTok/X for trending AI stories
  ├─ Searches YouTube for podcasts covering those stories
  ├─ Pulls transcripts from the best videos
  ├─ Fetches video metadata (views, likes, comments via get_video_metadata)
  ├─ Computes engagement velocity (via score_engagement)
  ├─ Finds the strongest 30-60 second moments
  └─ Outputs a CLIP BRIEF with:
      - Timestamped URL (youtube.com/watch?v=XXX&t=308)
      - Transcript quote
      - Category tag
      - Engagement analytics (views, likes, view velocity, like ratio)
      - 7-dimension scores
      - Brand gate result
      - Caption for Instagram
```

### Phase 2: Human Reviews
Editor receives the clip brief. Options: APPROVE → Phase 3, SHELVE → skip, MODIFY → revise.

### Phase 3: Human Produces (Editor Steps)
```
1. Click the timestamped URL → video opens at exact clip start
2. Screen record the clip (QuickTime, OBS, or built-in recorder)
3. Add text overlay (key quote on screen)
4. Add subtitles (auto-generate then tweak)
5. Add visual elements (waveform, dark background, channel credit)
6. Export as vertical reel
7. Post with the caption Goblin Recon wrote
```

### How to Watch the Clip
The clip brief includes a URL like `youtube.com/watch?v=ABC&t=308`. The `&t=308` tells YouTube: "start at 308 seconds." Click it → video plays from the exact moment. No scrubbing. No searching.

### What Goblin Recon Does vs Does NOT Do
| Does | Does NOT Do |
|------|-------------|
| Find what's trending | Download videos |
| Find the best podcast/video | Screen record |
| Find the exact 30-60s moment | Add text overlay |
| Check brand gate | Add subtitles |
| Write the Instagram caption | Export reels |
| Tag by category | Post to Instagram |
| Give cut instructions to editors | |

**Goblin Recon is the brain. The editor is the hands.**

Full process reference: `references/clip-mine-process.md`

## Output Format

Every report MUST lead with `## Decision` — recommended action in the first 3 seconds. Follow templates:
- Social Pulse and trend reports → `templates/social-pulse-report.md`
- Clip briefs → `templates/clip-mine-brief.md`
- Competitor reports → `templates/competitor-report.md`
- Content briefs → use `templates/social-pulse-report.md` unless the user asks for a standalone planning brief

### Auto-Save Rule

After every Social Pulse report, Fast Scan, Deep Social Scan, Signal Scan, Competitor report, or Clip Brief:
- Write the full output to `vault/reports/YYYY-MM-DD-{type}.md`
- Tell the user: `Saved -> vault/reports/<filename>`
- Never require the user to ask for a save

### Trend Report Must Include
- What's working on Instagram (formats, hooks, creators)
- What's on TikTok (formats, sounds, viral signals)
- News validation (tech news confirmation with URLs and dates)

### Clip Brief Must Include
- Decision (approve/shelve/modify)
- Background (2-3 sentences explaining source, speaker, and why the moment matters)
- Video metadata (title, channel, views, URL)
- Engagement analytics (views, likes, view velocity, like ratio, comment count)
- The moment text with exact timestamps
- Why post
- Scores by dimension (including scroll_stop)
- Brand gate result (angle, alignment score, blacklist violations)
- Platform variants (Instagram Reel, LinkedIn, YouTube Shorts)
- Fallback angle if rejected

## Winning Reel Formats (7 formats identified)

| Format | Hook | Best For |
|--------|------|----------|
| X vs Y comparison | "Which AI is most [X]?" | Model comparisons, tool face-offs |
| Shocking stat | "[N]% of [thing] is now [fact]" | Data stories, industry shifts |
| "They don't want you to know" | "How [group] is secretly [action]" | Exposé, manipulation stories |
| Celebrity controversy | "[Famous person] just [AI action]" | Hollywood, big names |
| Mind-blowing science | "This [thing] has no [X] but can [Y]" | Robotics, breakthroughs |
| Student vs system | "[Person] used [AI] and made [result]" | Democratization, tool tutorials |
| Quote card | "[Authority]: '[provocative quote]'" | CEO interviews, predictions |

## Testing Workflow (End-to-End)

1. **Layer 1** — Browser-based IG creator scan + TikTok hashtags + news sites
2. **Score stories** — Apply social_velocity first, then remaining dimensions. Confirm all >60.
3. **Layer 2** — Browser-based YouTube/IG/TikTok search for top 2–3 stories
4. **Pick best source** — Prioritize podcast/interview, English captions available, high scroll_stop
5. **Layer 3** — Extract transcript, find best moment, validate with extract_clip.py
6. **Collect engagement data** — Fetch video metadata (views, likes, comments) via get_video_metadata, compute velocity via score_engagement
7. **Brand gate** — Check blacklist, nuance words, brand angle. Score ≥8/15.
8. **Clip brief** — Follow template with platform variants, fill Engagement Analytics section

First test run (June 6, 2026): 5 stories found, 43-second clip extracted, full pipeline ~7 min. See `references/pipeline-test-jun-06-2026.md`.

## Session Discipline

For personal/current/future change notes, update `personal-dumpground/SESSION_LOG.md`. That folder is local-only and ignored by Git. Format:
```
## Session N — [Date]
### Changes Made
| File | Change | Reason |
### Test Results
| Test | Result |
### Open Items
- [ ] ...
```

## Pitfalls

### Search Stop Rule
If a named topic returns zero relevant results after 3 different search queries across 2+ platforms, stop searching and ask the user for a URL, screenshot, creator name, or more context. Do not keep trying loosely related keywords.

### YouTube Cookie Walls
YouTube may show a "Before you continue" consent dialog. Click the visible reject/accept option if available, then retry the original URL or query once. Do not use cookies or personal accounts to bypass access controls.

### Browser Stability
Avoid opening 3+ browser tabs simultaneously during scans. Use terminal/web extraction for stable public sources like Hacker News and reserve browser navigation for sources that need visual inspection.

### Public Social Access Limits
Reddit often returns a JS challenge without approved API access. Instagram may show login walls even for public profiles. TikTok may expose tag volume but hide individual videos behind login. Mark blocked sources as `access_status: blocked` and ask for manual assisted input only when the missing data is essential.

### English-Only Captions
The `get_youtube_transcript.py` script may return non-English captions. GenX brand rules require English-only outward content. Always check language before committing. Fall back to alternative sources.

### Instagram Scraping Is Fragile
Cookie walls and login gates block some accounts. Works for public profiles but unreliable at scale. The Meta API (disabled in config) would fix this. Until then, accept manual IG monitoring as fallback.

### Do Not Guess URLs
Do not invent article URLs from headlines or slugs. Extract real `href` values from category pages, search pages, feeds, sitemaps, or approved APIs. If a URL returns 404, retry once only by extracting the actual link from an index/search page, then move on.

For YouTube search result pages, extract real links from page anchors instead of constructing URLs from titles. Example browser-console pattern:

```js
Array.from(document.querySelectorAll('a[href*="watch?v="]'))
  .map((a) => ({ title: a.textContent.trim(), url: a.href }))
  .filter((item) => item.title && item.url)
```

### Blocked Sources Get One Confirmation Attempt
If a source returns a block page, captcha, DataDome/JS challenge, login wall, or rate-limit response, confirm once, set `access_status: blocked`, and move on. Do not spend repeated calls trying alternate scraping patterns.

### The Verge Needs Broad Link Extraction
The Verge article pages may not appear under narrow dated selectors. Use broad image-backed link extraction before declaring failure:

```js
Array.from(document.querySelectorAll('main > div > a[href]'))
  .filter((a) => a.querySelector('img') && a.href.includes('theverge.com'))
  .map((a) => ({ title: a.textContent.trim().slice(0, 70), url: a.href }))
```

### MCPs Are Optional
The entire pipeline runs on Hermes built-in tools (browser, web, terminal). The MCPs in `mcp.json` (memory, fetch, ghost-browser) are supplementary. The only one worth enabling early is `memory` for persisting brand-gate decisions. Ghost-browser is redundant with Hermes' built-in browser.

### Competitor Config Is Empty
`config/competitors.yaml` has zero entries. `run competitor scan` produces nothing until filled.

### Some YouTube Channel IDs Incomplete
Lex Fridman ID was truncated (fixed in Session 1). AI Exchange and AI Explained still have empty strings.

### delegate_task Subagents Don't Share Context
For news-site scanning, use direct browser/web extraction as the primary data collector. Reserve `delegate_task` for post-processing such as scoring, cross-referencing, and brand gate checks. If you do delegate, pass all necessary context explicitly: source URLs, search queries, story topics, IG creator handles, retry limits, and blocked-source rules.

### Tests Require PYTHONPATH
```bash
cd goblin-recon  # or wherever you cloned the project
PYTHONPATH=. .venv/bin/python -m unittest tests.test_scripts -v
```

### setup.sh Handles Full Setup
The `scripts/setup.sh` handles the complete setup: profile creation, SOUL.md installation, skill installation, auto-load configuration, Python venv, and verification. One command, done.

### Model Choice Matters
Layer 1 benefits from a capable model for multi-site scraping + scoring. Use the strongest approved provider/model available for full scans; use lighter approved models for bulk drafts.

### Never Hardcode User Paths — This Is Public, Not Personal
Goblin Recon is distributed to other users. Every path in SOUL.md, SKILL.md, AGENTS.md, setup scripts, and config references must be **project-root-relative** or use `~/.hermes/` (the one portable Hermes path).

❌ BANNED: any absolute user-home path, such as a local Desktop, Documents, or home-directory path.
✅ REQUIRED: `goblin-recon/` (relative from project root), `~/.hermes/profiles/goblin-recon/` (portable Hermes path), `./scripts/...`, `SOUL.md` (same directory).

**Test rule:** If a new user clones the repo into `~/Documents/` instead of `~/Desktop/`, every command in every file must still work. Zero find-and-replace-your-name steps. If you add a path to any file, ask: "Would this break if someone cloned this into a different directory?"

This rule was enforced in Session 3 after 6 hardcoded paths were found across SOUL.md and SKILL.md.

## Project Memory Files

Load before producing output:
- `memory/brand-rules.md` — Brand architecture, mission spine, B2C/B2B rules, audience profiles
- `memory/trend-history.md` — Previous trend scan results (avoid repeats)
- `memory/competitor-snapshots.md` — Previous competitor scan results
- `memory/content-performance.md` — Live content performance for scoring improvement

## Related Skills

- `genx-market-researcher` — Market research persona
- `genx-truth-teller` — Quality gate for GenX marketing outputs
- `genx-copy-chief` — Copywriting for GenX Academy (clip captions, platform variants)
- `caption-tone` — Single reusable caption-writing skill for platform-specific caption/description tasks
- `email-hook` — Reusable outbound email hook and short draft skill with automated quality gate scoring
- `competitor-profiling` — Cherry-picked for competitor scan research
- `social-content` — Cherry-picked for IG/TikTok platform variants and format analysis
- `copywriting` — Cherry-picked for caption writing
- `content-strategy` — Cherry-picked for weekly content planning
- `marketing-psychology` — Cherry-picked for engagement mechanics

## Reference Files

- `references/soul-md-example.md` — Canonical SOUL.md for goblin-recon profile
- `references/pipeline-test-jun-06-2026.md` — First end-to-end test results
- `references/session-2-bifurcation.md` — Session 2 bifurcation details
- `references/clip-mine-process.md` — Full Clip Mine workflow: AI discovery → scoring → editor production