--- name: rotate-scraping-proxies locale: caveman-lite source_locale: en source_commit: 82c77053 translator: "Julius Brussee homage — caveman" translation_date: "2026-05-03" description: > Escalate blocked scraping campaigns with provider-neutral proxy rotation — pick datacenter/residential/mobile pools, integrate rotation with scrapling, configure session stickiness for stateful flows, monitor cost and health, stay inside legal and ethical limits. Use after `headless-web-scraping` client-side stealth (StealthyFetcher, rate limiting, robots.txt) is insufficient and traffic is legitimate. license: MIT allowed-tools: Bash Read Write Grep Glob metadata: author: Philipp Thoss version: "1.0" domain: web-scraping complexity: intermediate language: Python tags: web-scraping, proxies, rotation, residential, scrapling, networking --- # Rotate Scraping Proxies Network-layer escalation for scraping where client-side stealth is exhausted. Proxy rotation is a last resort, not a default — expensive, ethically charged, and easily misused. This skill teaches when *not* to use it as much as how. ## When to Use - `headless-web-scraping` (Fetcher → StealthyFetcher → DynamicFetcher) tried and target still returns 403/429/geo-blocks - Rate limiting already at 3+ second intervals and `robots.txt` permits the path - User-Agent and TLS fingerprint already realistic (not default `python-requests`) - Scraping is legitimate: public data, no auth circumvention, no paywall bypass, no personal data harvested without legal basis - You can budget proxy traffic and accept operational complexity **Do not use** when: a public API exists (use it), the site's ToS forbids automated access, you would circumvent geo-licensing, or the goal is fraud / credential stuffing / sneaker bots / content piracy. ## Inputs - **Required**: Target URLs and the legal basis for scraping them - **Required**: Proxy pool credentials (read from environment, never hard-coded) - **Required**: Pool type — datacenter, residential, or mobile - **Optional**: Geographic targeting (country / region / city) - **Optional**: Rotation granularity — per-request (default) or sticky session - **Optional**: Daily traffic / spend cap - **Optional**: Rate limit delay in seconds (default: 1, even with rotation) ## Procedure ### Step 1: Pre-flight Legality and Ethics Check Gate the workflow on a documented legal and ethical review. Skipping this is the single biggest source of harm. ```python # Inputs to confirm before writing any code: # 1. Is the data public (no login required)? # 2. Does robots.txt permit the path? # 3. Does the site's ToS prohibit automated access? (read it) # 4. Would the scraping process personal data? If yes, what is the legal basis? # 5. Could this access circumvent geo-licensing, paywalls, or auth? # 6. Is there a public API or data dump that would make scraping unnecessary? # 7. Have you contacted the site owner if scope is large? ``` **Got:** Every question has a defensible written answer. The first "no" or "unknown" stops the procedure until resolved. **If fail:** - ToS forbids automated access — do not proceed; contact the site owner or use an official API or licensed dataset - Personal data with no legal basis — do not proceed; engage privacy counsel - Circumvents auth or geo-licensing — do not proceed under any circumstances ### Step 2: Choose a Pool Type Different pool types have different cost, detectability, and ethics. Pick the cheapest tier that solves your block. | Pool type | Detectability | Cost | Best for | |-----------|---------------|------|----------| | Datacenter | High (easily blocked by Cloudflare/Akamai) | $ | Sites with no real anti-bot, geo-shifting only | | Residential | Low (real ISP IPs) | $$$ | Sites that block datacenter ASNs | | Mobile | Very low (carrier-grade NAT, shared with thousands) | $$$$ | Sites that even block residential (rare) | **Ethical caveat for residential and mobile:** these pools route your traffic through real consumer connections. Operator consent models vary — some pay users, some bundle exit-node consent into "free VPN" EULAs that users do not read. Prefer providers with audited, opt-in consent. If you would not be comfortable with a stranger sending scraping traffic through your home router, do not send yours through theirs. **Got:** A documented choice with the cheapest viable tier and a brief note on why higher tiers were rejected (or why a higher tier is needed). **If fail:** - Datacenter blocked but residential over budget — narrow scraping scope (fewer URLs, slower cadence) before upgrading the tier - Cannot find a provider with documented opt-in consent — reconsider whether the scraping is necessary ### Step 3: Integrate Rotation with Scrapling Wire the proxy into scrapling fetchers. Read credentials from environment variables — never hard-code, never commit a `.env`. ```python import os import random from scrapling import Fetcher, StealthyFetcher # Pattern A: provider-managed rotating endpoint (one URL, provider rotates per request) PROXY_URL = os.environ["SCRAPING_PROXY_URL"] # http://user:pass@gateway.example:7777 fetcher = StealthyFetcher() fetcher.configure( headless=True, timeout=60, network_idle=True, proxy=PROXY_URL, ) # Pattern B: explicit pool, rotate yourself POOL = os.environ["SCRAPING_PROXY_POOL"].split(",") # comma-separated URLs def fetch_with_rotation(url): proxy = random.choice(POOL) fetcher = StealthyFetcher() fetcher.configure(headless=True, timeout=60, proxy=proxy) return fetcher.get(url) ``` **Got:** Requests succeed and the egress IP varies between calls. Confirm by hitting an IP-echo endpoint (e.g. `https://api.ipify.org`) before running the real scrape. **If fail:** - 407 Proxy Authentication Required — credentials wrong or password URL-encoding broke (re-encode special characters) - Same IP on every call — provider endpoint may be sticky by default; check docs for a `-rotating` or per-request flag - Massive latency increase — expected; rotation adds 200–2000ms per request ### Step 4: Sticky Sessions and Pool Health Decide rotation granularity per workload, then keep the pool healthy. ```python # Sticky session for stateful flows (login, multi-page checkout-like crawls) # Most providers expose a session ID via the username: # user-session-abc123:pass@gateway.example:7777 # All requests with the same session ID exit through the same IP for ~10 min. # Per-request rotation for anonymous bulk scraping (default) # Pool health check — call before bulk run def check_pool(pool, sample_size=5): sample = random.sample(pool, min(sample_size, len(pool))) alive = [] for proxy in sample: try: r = StealthyFetcher().configure(proxy=proxy, timeout=10).get( "https://api.ipify.org" ) if r.status == 200: alive.append(proxy) except Exception: pass return alive # Backoff on transient proxy failures def fetch_with_backoff(url, max_attempts=3): for attempt in range(max_attempts): try: r = fetch_with_rotation(url) if r.status not in (407, 502, 503): return r except Exception: pass time.sleep(2 ** attempt) return None ``` **Got:** Stateful flows preserve cookies across requests; bulk anonymous scraping shows IP variance across requests; dead proxies skipped instead of looping. **If fail:** - Login breaks mid-flow — rotation happening inside the session; switch to sticky-session credentials - All proxies in sample fail health check — pool exhausted or credentials expired; rotate credentials or contact provider ### Step 5: Monitoring, Cost Control, and Kill Switch Proxy traffic has a per-GB cost and a per-request cost. Runaway scrapers generate runaway invoices. Always include limits and an abort. ```python import time class ScrapeBudget: def __init__(self, max_requests, max_duration_seconds, max_failures): self.max_requests = max_requests self.max_duration = max_duration_seconds self.max_failures = max_failures self.requests = 0 self.failures = 0 self.start = time.monotonic() def allow(self): if self.requests >= self.max_requests: return False, "request cap reached" if time.monotonic() - self.start >= self.max_duration: return False, "time cap reached" if self.failures >= self.max_failures: return False, "failure cap reached (circuit breaker)" return True, None def record(self, success): self.requests += 1 if not success: self.failures += 1 budget = ScrapeBudget(max_requests=1000, max_duration_seconds=3600, max_failures=20) for url in target_urls: ok, reason = budget.allow() if not ok: print(f"Aborting: {reason}") break response = fetch_with_backoff(url) budget.record(success=response is not None) time.sleep(1) # rate limiting still applies even with rotation ``` **Got:** Budget caps trigger before runaway cost. Logs show per-proxy success rate so a bad egress IP can be identified and excluded. **If fail:** - Failure rate climbs above 20% — pause; the site has detected the rotation pattern (e.g. all your IPs share a subnet); switch pool type or stop - Cost-per-record exceeds expectations by 5x — cache aggressively, deduplicate URLs, batch where possible ## Validation - [ ] Step 1 legality check is documented in writing before any code runs - [ ] No proxy credentials, pool URLs, or session IDs in tracked files (grep for `gateway.`, `proxy=`, the provider hostname) - [ ] `.env` (or equivalent) is in `.gitignore` - [ ] Pool choice justified: cheapest viable tier, with consent model verified for residential/mobile - [ ] IP variance confirmed against an echo endpoint before the real run - [ ] Stateful flows use sticky sessions; bulk anonymous use per-request - [ ] Budget caps (requests, duration, failures) wired and tested - [ ] Rate limiting (≥1s) preserved — rotation is not an excuse to flood - [ ] `robots.txt` still respected — rotation does not override it ## Pitfalls - **Rotating before stealth is exhausted**: the site often does not need a new IP — it needs a realistic User-Agent, TLS fingerprint, and slower cadence. Try `StealthyFetcher` and rate limiting first; rotation is expensive and unethical to deploy unnecessarily. - **Hard-coded credentials**: pasting the proxy URL into source leaks it to git, container images, and stack traces. Read from environment variables or a secrets manager. - **Rotating mid-session**: per-request rotation breaks any flow that depends on cookies, CSRF tokens, or shopping-cart state. Use sticky sessions for stateful work. - **Treating rotation as "ethical anonymity"**: rotation hides *you* from the target, but does not make harmful scraping ethical. ToS, copyright, privacy law, and rate-limit ethics still apply unchanged. - **Using residential proxies for high-risk activity**: credential stuffing, sneaker botting, geo-pirating streaming content, fraud — explicitly out of scope. If your use case looks like this, stop. - **Ignoring `robots.txt` because "we have rotation now"**: rotation does not grant permission. The directive is the directive. - **No kill switch**: an unsupervised loop on a metered proxy pool turns into a four-figure invoice overnight. Always cap requests, duration, and failures. - **Choosing a residential pool with opaque consent**: some providers source exit nodes from "free VPN" EULAs that real users never read. Pay the premium for an audited, opt-in consent model. ## Related Skills - [headless-web-scraping](../headless-web-scraping/SKILL.md) — parent skill; always start there. Use this skill only as escalation. - [use-graphql-api](../use-graphql-api/SKILL.md) — prefer official APIs to scraping when one exists. - [deploy-searxng](../deploy-searxng/SKILL.md) — self-hosted search avoids scraping search engines entirely. - [configure-reverse-proxy](../configure-reverse-proxy/SKILL.md) — opposite network direction (serving instead of fetching), useful neighbor reference. - [security-audit-codebase](../security-audit-codebase/SKILL.md) — run after integrating credentials to confirm none leaked into the repo.