--- name: scraper-studio description: "Build and run AI-generated Bright Data scrapers from the terminal via `bdata scraper create` and `bdata scraper run`. Use this skill whenever the user wants to generate a scraper from a natural-language description, build a custom scraper without writing code, turn a URL + plain-English description into a reusable scraper, run an existing Bright Data collector against a URL, or batch-scrape a list of URLs through one collector. Triggers on phrases like 'build me a scraper for', 'create a scraper that extracts', 'generate a scraper from a description', 'turn this URL into a scraper', 'run this scraper on', 'run my collector', 'batch scrape', 'scrape these URLs', 'scrape a list of URLs', 'competitive pricing table', 'scraper studio', `scraper create`, `scraper run`, `--urls`, `--input-file`, `collector_id`, `automate_template`, or `/dca/`. Covers the AI flow (template create → trigger AI generation → poll progress), the single-URL run flow (async + poll by default, `--sync` for fast pages), the multi-URL batch flow (`--urls` / `--input-file` → one `/dca/trigger` call with array body), and the silent auto-fallback to the batch endpoint when a URL expands past the realtime page limit. Requires the Bright Data CLI." --- # Bright Data — Scraper Studio Build a scraper from natural language, then run it. Two commands live in this skill: - **`bdata scraper create `** — describe what you want in plain English; Bright Data's AI Flow generates a scraper template and returns a `collector_id`. - **`bdata scraper run `** — run that collector (or any existing one from the Bright Data web UI) against a URL and get the extracted data back. **The bridge between the two is `collector_id`.** It is printed by `create` and consumed by `run`. Always save it. For pre-built scrapers on platforms like Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc., **stop and use the [`data-feeds`](../data-feeds/SKILL.md) skill instead** — those scrapers already exist, are faster, cheaper, and more reliable than building a new one. Use Scraper Studio when no pre-built scraper covers the target site, or when the user wants a custom shape of output for an existing platform. ## Setup gate (run first) ```bash if ! command -v bdata >/dev/null 2>&1; then echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md" elif ! bdata zones >/dev/null 2>&1; then echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)" fi ``` Halt and route to setup if either check fails. Both commands require an authenticated CLI. ## Pick your path | Situation | Action | |---|---| | User describes data they want from a URL, no scraper exists yet | `bdata scraper create ""` → save the `collector_id` | | User has a `collector_id` and wants data from one URL | `bdata scraper run ` (default async + poll) | | User has a `collector_id` and wants data from many URLs | `bdata scraper run --urls "u1,u2,..."` or `--input-file urls.txt` (single batch call) | | Page is small and you want fast feedback (≤ ~50 s, single URL) | `bdata scraper run … --sync` | | Scraper ran but returned wrong / empty / partial data | inspect the output, then `bdata scraper heal ""` → review preview → approve → re-run to verify | | Site is a known platform (Amazon, LinkedIn, TikTok, …) | **stop — use `data-feeds` skill** | | You want SERP / discovery, not extraction | **use `search` skill** | | You want a one-off raw page fetch | **use `scrape` skill** | --- ## Action 1 — `scraper create` Generate a scraper from a URL + plain-English description. ```bash bdata scraper create "" [--name ] \ [--deliver-webhook ] [--timeout ] \ [--json | --pretty] [-o ] [--timing] [-k ] ``` The description is the most important input. A good description names every field you want and any conditions on how to find them. See [references/prompts.md](references/prompts.md) for examples of strong vs. weak descriptions. ```bash # Minimal bdata scraper create https://example.com/product/1 \ "Extract title, price, currency, image URL, and availability \ from this product page. If the price has a strike-through \ original price, capture both as price and original_price." # Save the full AI output for inspection bdata scraper create https://example.com/product/1 \ "Extract title, price, and image URL" \ --name product-scraper-v1 \ --pretty -o create.json ``` ### What happens under the hood `create` chains three Bright Data API calls — surface this to the user so they can debug from logs: 1. **`POST /dca/collector`** — creates an empty scraper template with a stub webhook delivery target (`https://example.com/webhook` by default). Returns a `collector_id` like `c_mp3tuab31lswoxvpws`. 2. **`POST /dca/collectors/{collector_id}/automate_template`** — triggers Bright Data's AI Flow with the description + URL. 3. **`GET .../automate_template/progress`** (polled) — waits for `status: "done"`. Generation typically takes **5–10 minutes** for moderately complex pages. ### Critical: hold the `collector_id` Every failure path in `create` (AI trigger fails, polling times out, generation finishes with `status: "failed"`) **still leaves a partially-built collector** at the printed `collector_id`. Always tell the user the id is recoverable — they can: - Open `https://brightdata.com/cp/scrapers/{collector_id}` to finish or inspect it in the web UI. - Re-trigger generation programmatically against the same id. - Delete it from the UI if they want a clean slate. Never claim "create failed, start over" without surfacing the `collector_id` from the response. ### `--timeout` — default 600 s AI generation can run 5–10 min for complex pages. If the page is simple, the default is plenty. For an elaborate description on a heavy site, raise it: ```bash bdata scraper create https://complex-site.com/page \ "Extract all 30 fields …" \ --timeout 1200 ``` On `Timeout after N seconds waiting for AI generation`, the `collector_id` is still printed. Re-check progress in the web UI rather than re-running `create` (which builds a new collector). ### `--deliver-webhook` — placeholder by default The CLI sets a stub webhook (`https://example.com/webhook`). This satisfies the API contract but **does not deliver anything**. For CLI use, leave the stub — you'll fetch results synchronously via `scraper run`. The real delivery target can be reconfigured in the [Bright Data web UI](https://brightdata.com/cp/scrapers) if the user wants webhook delivery for production runs. --- ## Action 2 — `scraper run` Execute a scraper against one or more URLs. ```bash bdata scraper run [url] \ [--urls "u1,u2,..." | --input-file ] \ [--sync [--sync-timeout 25-50]] \ [--timeout ] [--name ] [--version ] \ [--json | --pretty] [-o ] [--timing] [-k ] ``` **Pick exactly one input source:** positional ``, `--urls`, or `--input-file`. Combining them errors with `only one input source`. Multi-URL routes through `/dca/trigger` as a single API call with an array body — the canonical pattern from the [Scraper Studio Node](https://github.com/brightdata/bright-data-scraper-studio-nodejs-project) and [Python](https://github.com/brightdata/bright-data-scraper-studio-python-project) reference projects (`triggerWithUrls` / `trigger_with_urls`). One snapshot, one poll loop, one merged result array. **Do not hand-roll a `for url in $(cat urls.txt); do bdata scraper run ...` loop** — that's N API calls for what should be one. ### Choosing the run mode ``` ┌─────────────────────────────────────────────────┐ │ Expected to finish in ≤ ~50 seconds? │ └─────────────────────────────────────────────────┘ yes no │ │ ▼ ▼ ┌──────────────────────┐ ┌────────────────────────┐ │ --sync │ │ default (no flag) │ │ /dca/crawl │ │ trigger_immediate + │ │ one-shot, 25–50 s │ │ poll get_result │ └──────────────────────┘ └────────────────────────┘ │ │ on 202 timeout: response_id is printed — │ re-run WITHOUT --sync to poll for it ▼ (cleanly fall back to async) ``` | Mode | Endpoint | When to use | |---|---|---| | **Default (single URL, async + poll)** | `/dca/trigger_immediate` → poll `/dca/get_result` | Anything you expect to take more than ~50 s, anything paginated, anything you're not sure about. This is the safe default for one URL. | | **`--sync` (single URL)** | `/dca/crawl` | Single-page extractions you expect to complete in under 50 s. Faster path: one request, no polling. **Incompatible with multi-URL** — `/dca/crawl` accepts only one URL. | | **Multi-URL (`--urls` / `--input-file`)** | `/dca/trigger` (array body) → poll `/dca/dataset` | 2+ URLs through the same collector. One API call, one snapshot ID, one merged result array. Use the longer `--timeout` (default 3600s) for big batches. | ```bash # Default — async + poll (recommended for most cases) bdata scraper run c_mp3tuab31lswoxvpws https://www.amazon.com/dp/B08N5WRWNW # Pretty-printed, saved to disk bdata scraper run c_mp3tuab31lswoxvpws https://www.amazon.com/dp/B08N5WRWNW \ --pretty -o product.json # Sync mode for a fast page bdata scraper run c_mp3tuab31lswoxvpws https://example.com/p/1 --sync # Sync with a tighter server timeout bdata scraper run c_mp3tuab31lswoxvpws https://example.com/p/1 \ --sync --sync-timeout 30 # Multi-URL via comma-separated list — one batch call bdata scraper run c_mp3tuab31lswoxvpws \ --urls "https://example.com/p/1,https://example.com/p/2,https://example.com/p/3" \ --pretty -o products.json # Multi-URL via file (one URL per line; # comments and blanks ignored) bdata scraper run c_mp3tuab31lswoxvpws --input-file urls.txt -o products.json # Multi-URL via JSON array echo '["https://example.com/p/1","https://example.com/p/2"]' > urls.json bdata scraper run c_mp3tuab31lswoxvpws --input-file urls.json ``` ### `--sync-timeout` is bounded to 25–50 Anything outside that range exits with `--sync-timeout must be between 25 and 50 seconds`. Default is 50. Use values < 50 only when you want to fail fast and fall back to async on slow pages. ### `--sync` timeout recovery If `/dca/crawl` server-side times out, it returns 202 with `{"error":"crawl_results_timeout","response_id":"r_late_..."}`. The CLI surfaces the `response_id` and exits with a "Re-run without --sync" message. Do exactly that — drop `--sync` and re-run the same command; the default async path will poll `/dca/get_result` for the existing `response_id`. **Do not** treat sync timeout as a hard failure. ### Silent auto-fallback to batch (paginated / large pages) When a single URL expands to more pages than the realtime job limit allows (paginated listings, infinite scroll, "all reviews" pages, etc.), the realtime endpoints return an error like: ``` Request generated 501 pages and exceeded realtime job limit of 51 pages ``` The CLI detects this and **automatically falls back** to the batch flow (`/dca/trigger` → poll `/dca/dataset`). A one-line notice is printed. No flag is needed. Batch jobs use a longer poll interval (10 s) and a longer default timeout (1 hour). If the user only sees the notice and a long wait, that is expected — large jobs take time. Do not "fix" the fallback by switching APIs or restructuring the scraper; let it run. ### `--name` and `--version` - `--name ` — tag this run in the Bright Data dashboard. Useful when you're scripting many runs and want to find one later. - `--version ` — pin a specific scraper version (commonly `dev` when iterating in the web UI). Omit to use the latest published version. --- ## Action 3 — `scraper heal` Fix an existing scraper **in place** when it ran but returned wrong, empty, or partial data. The scraper's `collector_id` stays the same — it is improved, not replaced. ```bash bdata scraper heal "" [--url ] \ [--timeout ] [--max-retries ] [--no-retry] \ [--json | --pretty] [-o ] [--timing] [-k ] ``` **You are the detector.** The CLI never decides on its own that a scraper is broken — you inspect the run output and decide. A heal is slow and billable, so only heal when the data is actually wrong, not just legitimately empty. The `` is required and is the most important input. Name exactly what is wrong and what the correct output should be: *"The price field returns null — the selector moved into a ``. Capture price and currency again."* Vague prompts ("fix it") produce vague heals. The prompt is capped at 1000 characters. ### What happens under the hood 1. **`POST /dca/collectors/{collector_id}/refactor_template`** with `{prompt, custom_input: []}` — triggers the AI self-healing job. 2. **`GET .../refactor_template/progress`** (polled) — waits for `status: "done"`, same job shape and timing as `automate_template`. ### Output + the verify loop `heal` (without `--auto-approve`) usually ends at an **approval gate** rather than completing immediately. The response carries `status: "awaiting_approval"` with two key fields: - `preview_result` — sample rows the fixed scraper would produce, so you can judge whether the fix is correct before committing it. - `diff_summary` — a summary of what changed in the template. - `next_step` — points at `bdata scraper approve ` to commit (or `--reject` to discard). ```json { "collector_id": "c_mp3tuab31lswoxvpws", "status": "awaiting_approval", "preview_result": [{"price": "29.99", "currency": "USD"}], "diff_summary": "Updated price selector from .price to span[data-testid='price']", "next_step": "bdata scraper approve c_mp3tuab31lswoxvpws" } ``` Review `preview_result`, then run `bdata scraper approve ` to commit the fix — or pass `--reject` to discard it and re-heal with a sharper prompt. `approve` polls to `done` and hands back a `next_step` = `bdata scraper run ` to verify. Pass `--url ` to `heal` so the approve and run steps are concrete. The full self-healing loop is now: ``` run → inspect → heal → review preview → approve → run → verify ``` To skip the gate entirely, use `heal --auto-approve` — it approves automatically and polls through to `done`. ### Failure is non-destructive If a heal fails (429 cap exhausted, timeout, terminal `failed`), the existing scraper is **unchanged and still works** as it did before. The CLI says so and prints the `collector_id`. Unlike a failed `create`, nothing half-built is left behind. --- ## Action 4 — `scraper approve` A `scraper heal` (without `--auto-approve`) stops at an approval gate: `status: "awaiting_approval"`, with `preview_result` (sample rows the fixed scraper would produce) and a `diff_summary`. Review the preview, then commit the fix: ```bash bdata scraper approve [--reject] [--url ] \ [--timeout ] [--json | --pretty] [-o ] [-k ] ``` - Approves by default (`POST /dca/collectors/{id}/resume_automation_job {"message": true}`), then polls to `done` and hands back a `next_step` = `bdata scraper run ` to verify. - `--reject` sends `{"message": false}` to discard the proposed fix; re-heal with a sharper prompt to try again. - If a heal needs multiple approvals, `approve` may stop at `awaiting_approval` again — just run it again. `awaiting_approval` is **not** a failure — it means the fix is ready and waiting for your decision. --- ## Full create-then-run workflow Capture the `collector_id` cleanly with `jq`, then chain into `run` with the multi-URL batch path: ```bash # 1. Create the scraper, save the AI output, extract the collector_id bdata scraper create https://example.com/product/1 \ "Extract title, price, currency, image URL, and availability" \ --pretty -o create.json # 2. Pull the collector_id (or copy from the human-readable summary) COLLECTOR_ID=$(jq -r '.collector_id // .id' create.json) echo "Built: $COLLECTOR_ID" # 3. Run it on every URL in one batch — single API call, merged result array bdata scraper run "$COLLECTOR_ID" --input-file urls.txt \ --pretty -o out/results.json ``` **Do not** wrap `bdata scraper run` in a `for url in $(cat urls.txt); do ... done` loop — that's N API calls and N snapshots. Use `--input-file urls.txt` (or `--urls "..."`) instead; the CLI POSTs all of them to `/dca/trigger` in a single array body and returns one merged result array. For more end-to-end recipes (batch input file shapes, error recovery, web-UI handoff), see [references/recipes.md](references/recipes.md). --- ## Common mistakes 1. **Inventing command names.** The commands are exactly `bdata scraper create` and `bdata scraper run`. There is no `bdata generate`, no `bdata scrape-batch`, no `bdata data`, no `bdata build`. If you're tempted to use one of those, you're hallucinating — run `bdata scraper --help` to verify. 2. **Re-running `create` after a timeout.** Generation creates a fresh collector every time. If polling times out, the half-built collector is still printed in the error output. Resume it in the web UI or wait and re-poll — don't burn another collector. 3. **Defaulting to `--sync` for everything.** Sync caps at 50 s server-side. Any paginated page or heavy SPA will time out. Default async is the right choice unless you specifically know the page is small. 4. **Skipping Scraper Studio for known platforms.** If the target is Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc., the [`data-feeds`](../data-feeds/SKILL.md) skill exposes pre-built scrapers that are faster, cheaper, and more reliable. Only build a custom scraper when no pre-built one exists for the site, or when the user explicitly wants a custom output shape. 5. **Treating sync timeout as failure.** A `--sync` 202 timeout returns a valid `response_id` — the job is still running on the backend. Re-run the same command without `--sync` to pick it up. 6. **Throwing away `collector_id` / `response_id` on error.** The CLI prints them in every failure path on purpose. Both are recoverable via the API and the web UI. Always surface them to the user. 7. **Trying to disable the batch auto-fallback.** It has no flag. The fallback is the correct behavior when a URL expands past the realtime page limit. Let it run. 8. **Hand-rolling a `for url in $(cat urls.txt); do bdata scraper run ...` loop.** That's N API calls, N snapshot IDs, and N poll loops for what the API natively treats as one batch. Use `--input-file urls.txt` or `--urls "u1,u2,..."` — the CLI posts the whole array to `/dca/trigger` in a single request and returns one merged result array. This mirrors the canonical `triggerWithUrls` / `trigger_with_urls` helpers from the Scraper Studio reference SDKs. 9. **Vague descriptions in `create`.** A description like "scrape the page" produces a generic scraper. Name every field, name conditions ("if there's a sale price, capture both"), name disambiguators ("the price near the title, not in the recommendations sidebar"). See [references/prompts.md](references/prompts.md). 10. **Re-running `create` to fix a broken scraper.** That builds a *new* collector and orphans the old one. To fix an existing scraper, use `bdata scraper heal ""` — it mutates the scraper in place so your saved `collector_id` keeps working and improves. 11. **Treating `awaiting_approval` as a failure.** It is the normal end state of a heal — the fix is computed and waiting for your decision. Review `preview_result`, then `bdata scraper approve ` (or `--reject`). Use `heal --auto-approve` to skip the gate. --- ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | `bdata: command not found` | CLI not installed | See [`brightdata-cli`](../brightdata-cli/SKILL.md) for install. | | `Invalid or expired API key` | Not logged in | `bdata login` (or `bdata login --device` for SSH). | | `create` returns no `id` | API call failed before template was created | Check `--timing` output for the failing request; verify network and account status. | | `Timeout after 600 seconds waiting for AI generation` | Page is complex; default poll exceeded | The `collector_id` is still printed. Open it in the web UI; or re-run with `--timeout 1200`. | | `status: "failed"` from progress poll | AI Flow couldn't build the template | Improve the description — be more specific about fields and selectors. Try again with a cleaner URL (e.g. a canonical product page, not a search result). | | `--sync` 202 with `crawl_results_timeout` | Page took > sync server cap | Re-run **without** `--sync` to poll `/dca/get_result` for the printed `response_id`. | | `--sync-timeout must be between 25 and 50 seconds` | Out-of-range value | Use a value in `[25, 50]`. | | `Request generated N pages and exceeded realtime job limit` | URL expanded too far | This is **handled automatically** — wait for the batch fallback notice and let it poll. | | No data returned, just `[]` or `{}` | Selectors didn't match | The scraper template ran but extracted nothing. Open `https://brightdata.com/cp/scrapers/{collector_id}` to inspect/edit; or rebuild with a tighter description. | --- ## Reference files - **[references/prompts.md](references/prompts.md)** — How to write a description for `scraper create` that the AI Flow can act on. Examples of strong vs. weak descriptions, field-listing patterns, and conditional rules. - **[references/recipes.md](references/recipes.md)** — End-to-end recipes: capture `collector_id` from create, batch run a list of URLs, handle sync→async fallback, recover from a failed create, list scrapers from the dashboard. - **[references/api-flow.md](references/api-flow.md)** — Exact REST endpoints, payloads, and status sentinels the CLI uses (`/dca/collector`, `/dca/collectors/{id}/automate_template`, `/dca/trigger_immediate`, `/dca/get_result`, `/dca/crawl`, `/dca/trigger`, `/dca/dataset`). Read this when debugging unexpected CLI output or when the user wants to hit the API directly.