--- name: claude-watch description: Use when the user gives a video URL (YouTube, Instagram, Loom, TikTok, direct link) and wants Claude to actually SEE the video — a scene-by-scene / frame-by-frame breakdown, not just the transcript. Triggers on "/claude-watch ", "watch this video", "break this reel down", "analyze this reel". --- # claude-watch — see the video, not just the transcript Claude has no native video model. This skill splits a video into the two things Claude *does* understand — **images + text** — and reads both with timestamps aligned. Run every step in order. Never skip the dense-hook sampling in Step 2; that is the whole trick. --- ## Step 0 — Set up the workspace ```bash WATCH_DIR="${TMPDIR:-/tmp}/claude-watch" rm -rf "$WATCH_DIR" && mkdir -p "$WATCH_DIR" && cd "$WATCH_DIR" ``` Verify the three tools exist before doing anything else. If one is missing, tell the user which one and stop — don't silently degrade to a transcript-only answer: ```bash for t in yt-dlp ffmpeg ffprobe; do command -v "$t" >/dev/null || echo "MISSING: $t"; done ``` --- ## Step 1 — Download the video ```bash yt-dlp -o video.%\(ext\)s \ -f "bv*[height<=720]+ba/b[height<=720]/bv*+ba/b" --merge-output-format mp4 "" ffprobe -v error -show_entries format=duration -of csv=p=0 video.mp4 # duration in seconds ``` **Cap the height at 720.** Every frame gets downscaled to 200–360px anyway, so pulling 1080p/4K buys nothing and costs a lot: an unbounded `-f "bv*+ba/b"` on a 19-minute video pulled **516 MB** where 720p would have been a fraction of that. The trailing `/bv*+ba/b` is the fallback for sources that have no ≤720 rendition. If yt-dlp warns `No supported JavaScript runtime could be found`, YouTube extraction is running in a deprecated path and some formats will be missing. Install Deno (`winget install DenoLand.Deno`, or `brew install deno`) and make sure it's on PATH — yt-dlp picks it up automatically. If the download fails with a login/age wall, ask the user for a cookies file rather than guessing: `yt-dlp --cookies-from-browser chrome ...`. Never bypass a paywall or CAPTCHA. **Note the duration.** It decides the body sampling rate in Step 2. --- ## Step 2 — Two-stage frame sampling (the important part) The hook holds the most motion and decides retention, so sample it densely. The body is mostly held shots, so sample it sparsely. One frame every few seconds across the *whole* video only catches end-states — it misses word-by-word text reveals, fast cuts, and count-up animations. ### HOOK — first 15s at 15 fps, tiled into contact sheets ```bash mkdir -p hook ffmpeg -y -t 15 -i video.mp4 \ -vf "fps=15,scale=200:-1,tile=5x9:margin=6:padding=4:color=white" \ -an hook/sheet_%02d.jpg ``` 5 cols × 9 rows = **45 frames per sheet = 3.0 seconds per sheet**. 15s → 5 sheets. **Timestamp math:** on sheet `k` (1-indexed), cell `i` (1-indexed, reading left→right, top→bottom) is global frame `N = (k-1)*45 + i`, at **t = (N-1) / 15 seconds**. So sheet 2, cell 1 → N=46 → t = 3.00s. *(Verified by pixel-comparing the extracted cell against the source frame — the off-by-one alternative scores 11 dB worse.)* Two behaviors worth knowing, both confirmed: - **Partial sheets are kept.** A video shorter than 15s doesn't divide evenly into 45-frame sheets; ffmpeg flushes the remainder as a final short sheet rather than dropping it. A 7s clip yields 3 sheets, not 2 — no tail is silently lost. - **Expect ~130–210 KB per sheet** (roughly 1028×1061 for 16:9 source). A full 15s hook is ~900 KB across 5 sheets. Vertical/9:16 source produces taller, somewhat larger sheets. ### BODY — sparse pass across the whole video Pick the interval from the duration so you land in the 25–60 frame range: | Duration | Interval | ffmpeg `fps=` value | |---------------|-----------|---------------------| | under 2 min | 3.5s | `1/3.5` | | 2–5 min | 6s | `1/6` | | 5–15 min | 15s | `1/15` | | over 15 min | 30s | `1/30` | ```bash mkdir -p body ffmpeg -y -i video.mp4 -vf "fps=1/3.5,scale=360:-1" body/f_%03d.jpg # adjust fps= per table ``` Body frame `n` (1-indexed) is at **t ≈ (n - 0.5) × interval**. **Then tile the body frames too** — reading 38 separate images burns far more context than reading two sheets, and side-by-side placement makes b-roll density visible at a glance: ```bash ffmpeg -y -framerate 1 -i body/f_%03d.jpg \ -vf "scale=300:-1,tile=5x4:margin=6:padding=4:color=white" body_sheet_%02d.jpg ``` 20 frames per sheet. Body sheet `k`, cell `i` → frame `n = (k-1)*20 + i` → `t ≈ (n - 0.5) × interval`. ### Optional — cut detection (worth it for fast-cut edits) ```bash ffmpeg -i video.mp4 -filter:v "select='gt(scene,0.4)',showinfo" -f null - 2>&1 \ | grep showinfo | sed -n 's/.*pts_time:\([0-9.]*\).*/\1/p' ``` That prints the timestamp of every hard cut. Cut *cadence* — cuts per second in the hook vs the body — is one of the most useful numbers you can hand the user. --- ## Step 3 — Get the transcript **Prefer real captions when they exist** — free, exact, and already timestamped: ```bash yt-dlp --write-auto-subs --write-subs --sub-langs "en.*" --skip-download \ --convert-subs srt -o subs "" ``` Fall back to local transcription only if no captions came back: ```bash ffmpeg -y -i video.mp4 -ar 16000 -ac 1 audio.wav whisper-cli -m ~/.claude/models/ggml-base.en.bin -f audio.wav -osrt -of transcript ``` Use `-osrt` (not `-otxt`) — you need the timestamps to line the words up with the frames. **Always de-duplicate YouTube auto-captions before reading them.** Auto-captions are written as a *rolling* transcript: each cue repeats the previous line plus a few new words, to fake a scroll effect. A 19-minute video came back as 1100 cues that were mostly restatements of each other — reading that raw wastes enormous context for no added information. ```bash python3 - <<'PY' import re t = open('subs.en.srt', encoding='utf-8').read() blocks = re.findall(r'(\d\d:\d\d:\d\d),\d+ --> .*?\n(.*?)\n\n', t, re.S) seen, out = set(), [] for ts, txt in blocks: line = ' '.join(txt.split()) if line and line not in seen: seen.add(line); out.append(f'{ts} {line}') open('transcript_clean.txt', 'w', encoding='utf-8').write('\n'.join(out)) print(f'{len(blocks)} cues -> {len(out)} unique lines') PY ``` Read `transcript_clean.txt`, not the raw SRT. **Always sanity-check the transcript against the frames.** Local models mishear proper nouns (they'll render "Claude" as "plot", "n8n" as "any DNA"). On-screen text is ground truth; when the audio and the pixels disagree, the pixels win. --- ## Step 4 — Read frames + transcript together Read the hook sheets **in order**, then the body frames **in order**, with the SRT open beside them. For each moment, hold three things at once: what's being said, what's on screen, and what just changed from the previous frame. Then output: 1. **Corrected transcript** — the SRT, with names and jargon fixed against the on-screen text. 2. **Scene-by-scene breakdown** — a row per beat: `time | what's on screen | on-screen text (verbatim) | the cut into it` 3. **Why it works** — 2–3 *concrete* mechanisms, each tied to a timestamp. Not "great hook" — "0.0–1.2s: claim + number before any face appears; the first cut lands at 1.4s, before the scroll-away window closes." 4. **Steal-the-structure notes** — the reusable skeleton with beat durations, so the user can swap in their own content and keep the shape. Quote on-screen text **verbatim**, including line breaks and emoji. That text is usually the actual script; the voiceover is the gloss on it. --- ## Step 5 — Clean up ```bash rm -rf "${TMPDIR:-/tmp}/claude-watch" ``` Keep the files if the user asks for a second pass — re-downloading is the slow part. --- ## Notes - **Everything is local.** No API calls, no upload, no per-minute cost. - **Rough budget:** ~5 hook sheets + ~30 body frames is a few hundred KB of images. A 10-minute video is fine. A 2-hour video is not — ask the user for the segment they care about and add `-ss -t ` to the ffmpeg calls. - **Respect the source.** Only download videos the user is entitled to. Don't strip DRM, don't bypass paywalls or CAPTCHAs, and don't republish someone else's footage — analyze it.