--- name: cdp-graphql-scraper description: >- Scrape data from sites you're authorized to access by attaching Playwright to your OWN already-logged-in Chrome over the DevTools protocol (CDP) and harvesting the JSON/GraphQL responses the page already fetches — instead of re-authenticating, driving login forms, or hammering endpoints directly. Use this whenever a task involves collecting structured data from a logged-in web app, building a resilient web scraper/collector, intercepting XHR/GraphQL/API responses, or when an existing scraper is getting rate-limited, soft-blocked, or returning empty results. Also reach for it for the operational side — running, resuming, or debugging a long crawl, or making one survive blocks and restarts. Covers the anti-block hygiene (human-paced delays, batch cooldowns, block-signal detection, resumable state) that naive scrapers miss. --- # Logged-in CDP + response-interception scraper Most scraping breaks because it fights the site: it re-logs-in with brittle form automation, parses HTML that changes weekly, and fires requests fast enough to trip bot defenses. This skill teaches a calmer approach that survives far longer: **Attach to a browser you already control, and read what it's already saying.** You launch your normal Chrome (with your real, human-established session), Playwright connects to it over CDP without opening a new browser, and you simply *listen* to the JSON the site's own frontend fetches as you navigate. No login automation, no HTML parsing, no synthetic API calls the server can fingerprint. ## Authorized use only This technique is for data you have a legitimate right to collect: your own accounts, sites whose Terms of Service permit it, APIs you're licensed for, or data you've been explicitly authorized to gather. Before scraping a third-party service, check its ToS and `robots.txt`, and keep request volume low enough that you never degrade the service for others. Automated collection can also violate a platform's ToS and get the *account* banned even when the IP is fine — so use a throwaway/dedicated account for anything against a service you don't own, never a personal or important one. When in doubt, prefer an official/paid API. ## Why response interception beats DOM scraping A modern web app renders from JSON it fetches over `fetch`/XHR — often a GraphQL endpoint. That JSON is richer, more stable, and cheaper to parse than the DOM: - **Stable**: field names in an API payload change far less often than CSS/DOM. - **Complete**: the payload usually carries more than the page shows (counts, ids, flags, pagination cursors). - **Cheap**: no waiting on layout, no fragile selectors, no screenshot OCR. So the game is: navigate like a human, and catch the responses in a listener. ## Setup: attach to your own Chrome over CDP Launch Chrome with remote debugging **before** running the script, log in normally (as a human — solve any captcha yourself, once), and leave it open: ``` # Windows chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\chrome-debug" # macOS /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 ``` Verify it's reachable — this JSON means you're good: ``` curl http://127.0.0.1:9222/json/version ``` Then connect over CDP instead of launching a fresh browser — this reuses your real session, cookies, and fingerprint: ```python browser = await p.chromium.connect_over_cdp("http://127.0.0.1:9222") context = browser.contexts[0] # your existing window page = context.pages[0] # reuse an open tab page.on("response", handle_response) # <- the whole trick ``` A dedicated `--user-data-dir` keeps this debug profile separate from your day-to-day browsing. ## The interception pattern Register one response handler, filter to the endpoints you care about, and pull the fields out defensively (sites ship multiple response shapes simultaneously — old and new schema, logged-in vs. logged-out variants): ```python CAPTURED = {} # keyed by whatever id you're collecting async def handle_response(response): url = response.url if "/api/graphql" not in url and "/graphql" not in url: return try: text = await response.text() if not text.strip(): return if looks_blocked(text, url, response.status): print("🚨 block signal — stopping") return data = json.loads(text) except Exception: return # partial/binary/non-JSON — ignore, don't crash node = dig(data, ...) # tolerate several key paths, not just one if node: CAPTURED[node["id"]] = normalize(node) ``` Key habits: never let a single bad response throw; try old and new key paths before giving up; and treat an empty 200 as "gated," not "done." `scripts/cdp_scraper_template.py` is a complete, runnable skeleton of this — queue, listener, delays, resume, and output. Start from it and fill in the two site-specific hooks (`URL_MATCH` and `extract`). ## Anti-block hygiene (the part naive scrapers skip) A scraper that fires as fast as it can gets soft-blocked within minutes. Pace it like a person and it can run for hours: - **Human-paced delays** between navigations — randomized, not a fixed interval (e.g. `random.uniform(8, 20)` seconds). Fixed cadence is itself a bot tell. - **Batch cooldowns** — after every N targets, sleep 60s+. Bursts look robotic. - **Soft-block auto-stop** — if the expected response fails to appear several times in a row (e.g. 3 empty/absent captures), the site is likely throttling you. Stop *on your own* and resume later; pushing through deepens the block. - **Hard-block detection** — treat HTTP 429, `checkpoint_required`, `challenge` URLs, or "action blocked" text as a full stop and alert the user; these precede an account/IP suspension. - **One crawler at a time** — never run two scrapers against the same account concurrently. They share the session cookie, so the site sees double the rate and blocks you faster. If multiple agents/people share the repo, confirm who owns the run before starting. - **Protect the IP if needed** — for larger jobs, a rotating residential/mobile proxy spreads requests across IPs. (Rotation protects the *IP*, not the *account* — a burner account is still what protects the account.) ## Resumability (so a block or crash costs you nothing) Record each successfully-collected id to a small state file and skip it on the next run. Then "resume" is just "run the same command again": ```python done = set(json.load(open(DONE, encoding="utf-8"))) if os.path.exists(DONE) else set() queue = [h for h in targets if h not in done] # ... after each success: done.add(h); save(done) ``` Save after *every* success, and save again in a `finally:` block so a mid-run crash still persists progress. ⚠ **One subtle trap: don't corrupt the state file with a BOM.** On Windows, writing this JSON with PowerShell `Out-File`/`Set-Content` prepends a UTF-8 BOM, and a naive `json.load` then reads zero completed items — so the next run re-scrapes *everything* and hammers the site. Always write state with Python (`json.dump(..., open(path, "w", encoding="utf-8"))`), never a shell redirect. ## Common runtime pitfalls | Symptom | Cause → fix | |---|---| | Script dies instantly, no logs | The debug Chrome was closed → relaunch it, rerun (resume handles the rest) | | No output when run in background | Python buffered stdout → run with `python -u` | | Emoji/Unicode crash on Windows | cp949 console → run with `python -X utf8` (or `sys.stdout.reconfigure(encoding="utf-8")`) | | Whole queue re-runs from scratch | BOM in the state file (see above) → rewrite it with Python, not a shell | | Expected response never captured | Wrong URL filter, or the site gates that endpoint when logged-out → verify you're logged in and check the real URL in DevTools → Network | | Empty 200 responses | Soft-block or gating, not "no data" → back off and retry later | ## Reference files - `scripts/cdp_scraper_template.py` — a complete, generic scraper skeleton (CDP attach, response listener, human delays, batch cooldown, block detection, resumable state, JSONL output). Copy it and implement `URL_MATCH` + `extract`.