--- name: retail-provider-fix description: Diagnose and fix retail scraping failures for individual dealers — wrong prices, OOS, site structure changes. --- # Retail Provider Fix Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline. ## Where things live (current as of 2026-04) | Piece | Path / Location | | -------------------------------- | ------------------------------------------------------------------------------------------------- | | Extraction code | `devops/pollers/shared/price-extract.js` | | Per-provider config | `PROVIDER_CONFIG` in `price-extract.js` | | CF-clearance / Byparr sidecar | `devops/pollers/shared/cf-clearance.js` | | Goldback scraper | `devops/pollers/shared/goldback-scraper.js` | | Fly.io app | `devops/pollers/remote-poller/` (app `staktrakr`) | | Home poller (LXC) | `devops/pollers/home-poller/` | | Home dashboard / provider editor | `http://:3010` | | Cron source of truth | `devops/pollers/home-poller/docker-entrypoint.sh` | | providers.json source of truth | **sqld** tables `provider_coins` + `provider_vendors` — regenerated by `export-providers-json.js` | **Critical topology** (STAK-478): - **Home poller (LXC) is the sole retail + goldback scraper.** Retail cron `30 * * * *` (hourly at :30), goldback `5 16 * * *` daily. - **Fly.io `staktrakr` is a thin publisher.** `RETAIL_ENABLED=0`, `GOLDBACK_ENABLED=0`. Fly only runs spot, publish, serve.js. Do not debug retail on Fly. - Both pollers consume the same `PROVIDER_CONFIG` via `shared/price-extract.js`. ## Step 1 — Live diagnostic against the home poller Scrape the failing URL directly through the home poller's Firecrawl to see what the extractor actually sees. Get the URL from the home dashboard provider editor (`:3010`) for the failing coin/vendor row. ```bash # From the home LXC (or over Tailscale): docker exec -it staktrakr-home-poller curl -s -X POST http://firecrawl-api:3002/v1/scrape \ -H 'Content-Type: application/json' \ -d '{"url":"PROVIDER_URL_HERE","formats":["markdown"],"waitFor":6000,"onlyMainContent":false}' \ | python3 -c "import sys,json; d=json.load(sys.stdin); \ md=d.get('data',{}).get('markdown',''); \ w=d.get('data',{}).get('warning',''); \ print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])" ``` ## Step 2 — Interpret the output | Symptom | Meaning | Fix category | | -------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------- | | `warning: waitFor not supported` | Firecrawl image too old to honor waitFor | Upgrade Firecrawl image OR set `phase: "phase0"` (Playwright-direct renders JS natively) | | Markdown is only `Loading...` | Product table is JS-rendered | Switch `phase` to `cf-clearance-first` or `phase0` | | Metadata URL ≠ sourceURL (homepage redirect) | Bot detection (Cloudflare) | `phase: "cf-clearance-first"` + `cf_clearance_fallback: true` | | Spot prices appear before product price | `MARKDOWN_HEADER_SKIP_PATTERNS` not matching | Update header-skip regex for provider | | `fractional_weight — 1/4 oz` | Nav/menu fractional link matched before product | Set `fractionalExempt: true` OR tighten `MARKDOWN_CUTOFF_PATTERNS` | | `page loaded, no price` | Page rendered but `extractPrice` regex missed | Update extraction logic for provider | | `out_of_stock — PRE-ORDER` | Item actually on pre-order | Fine unless provider is in `PREORDER_TOLERANT_PROVIDERS` | ## Step 3 — Config map in `price-extract.js` All runtime behavior is driven by a single table, `PROVIDER_CONFIG`. Look up by provider id, not by line number. | Symbol | Purpose | | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PROVIDER_CONFIG[id]` | Per-provider overrides: `phase`, `waitFor`, `waitUntil`, `waitAfter`, `timeout`, `onlyMainContent`, `retryOn408`, `fractionalExempt`, `cf_clearance_fallback`, `proxy` | | `PROVIDER_DEFAULTS` | Defaults merged into every provider | | `providerCfg(id)` | Helper — returns merged config for a provider id | | `MARKDOWN_HEADER_SKIP_PATTERNS` | Per-provider regex — cuts everything BEFORE the match | | `MARKDOWN_CUTOFF_PATTERNS` | Per-provider regex array — cuts everything AFTER the first match | | `PREORDER_TOLERANT_PROVIDERS` | Providers where `PRE-ORDER` does NOT imply OOS | | `USES_AS_LOW_AS` | Providers that use "As Low As" as primary price (currently empty) | | `SLOW_PROVIDERS` / `FIRECRAWL_PREFERRED_PROVIDERS` / `PLAYWRIGHT_ONLY_PROVIDERS` | Back-compat shims auto-derived from `PROVIDER_CONFIG`. Do not edit directly — change `PROVIDER_CONFIG` instead. | **Never edit `providers.json` by hand.** It is regenerated from sqld every 5 min via `export-providers-json.js`. Edit URLs/selectors through the **home dashboard provider editor at `:3010`**, which calls `updateVendorUrl(client, coinSlug, vendorId, url)` in `provider-db.js`. ## Step 4 — Phase routing `phase` on each provider selects the scrape strategy. This is the primary knob — flip it before chasing regex/header changes. | `phase` | Order | | -------------------- | ----------------------------------------------------- | | `phase0` (default) | Playwright-direct → Firecrawl → CF-clearance fallback | | `firecrawl` | Skip Phase 0, Firecrawl → CF-clearance fallback | | `cf-clearance-first` | Byparr/CF-clearance sidecar → Firecrawl fallback | Current assignments in code: - `apmex`, `monumentmetals`, `herobullion`, `gainesvillecoins` → `firecrawl` (HTML pipe tables lose structure via innerText) - `jmbullion`, `bullionexchanges` → `cf-clearance-first` + `cf_clearance_fallback: true` (CF protected) - `sdbullion`, `summitmetals`, `goldback` → `phase0` defaults ## Step 5 — CF-protected vendors (JMBullion, BullionExchanges) Current solution is the **Byparr sidecar** reached via `phase: "cf-clearance-first"` and `cf_clearance_fallback: true`. This replaces the old residential-proxy / Tailscale-exit-node approach. - Byparr fetches the page with a valid `cf_clearance` cookie and returns the HTML. - If Byparr has no usable HTML, code launches Playwright with the `cf_clearance` cookie attached. - Fly-based BullionExchanges is ~93% on Fly's own IP (no proxy needed); home poller uses its residential IP. If CF bypass stops working: 1. `docker logs staktrakr-byparr` on the home LXC — confirm sidecar up. 2. Check `shared/cf-clearance.js` — cookie cache, retry count, timeout. 3. Verify `retryOn408` and `timeout` are generous enough (`bullionexchanges` currently `70_000`). ## Step 6 — Dry run a single vendor Home poller (primary path for retail): ```bash # On the home LXC docker exec -it staktrakr-home-poller bash cd /app DATA_DIR=/data \ FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \ PLAYWRIGHT_LAUNCH=1 \ COINS=ase PROVIDERS=jmbullion \ DRY_RUN=1 node price-extract.js # Multi-provider DATA_DIR=/data \ FIRECRAWL_BASE_URL=http://firecrawl-api:3002 \ PLAYWRIGHT_LAUNCH=1 \ COINS=ase,age PROVIDERS=jmbullion,bullionexchanges \ DRY_RUN=1 node price-extract.js ``` Fly (only if `RETAIL_ENABLED=1` is being re-enabled or validated): ```bash fly ssh console -a staktrakr cd /app DATA_DIR=/data/staktrakr-api-export/data \ FIRECRAWL_BASE_URL=http://localhost:3002 \ PLAYWRIGHT_LAUNCH=1 \ COINS=ase PROVIDERS=jmbullion \ DRY_RUN=1 node price-extract.js ``` Success: `✓ jmbullion: $XXX.XX (cf-clearance)` / `(firecrawl)` / `(playwright)`. ## Step 7 — Ship the fix - **Code change in `shared/price-extract.js` or `shared/cf-clearance.js`** — worktree → PR to `dev`. Home poller rebuilds its image and restarts to pick it up. Fly picks it up on next `fly deploy -a staktrakr`. - **URL / selector change** — edit in home dashboard provider editor (`:3010`). Writes to sqld. `export-providers-json.js` regenerates `providers.json` within 5 min and publishes on next Fly `run-publish` cycle (`8,23,38,53 * * * *`). - **Threshold / cron tweak** — edit `devops/pollers/home-poller/docker-entrypoint.sh` (authoritative) and/or `devops/pollers/remote-poller/docker-entrypoint.sh`. ## Gap-fill / FindBullionPrices (FBP) FBP is now a **gap-fill pass**, not a primary provider. `devops/pollers/home-poller/run-fbp.sh` runs with `PATCH_GAPS=1`, queries sqld for today's failed vendors, scrapes FBP for those specific coins, writes recovered prices. There is no longer a `FBP_DEALER_NAME_MAP` in `price-extract.js` — dealer resolution happens via FBP's own JSON joined against sqld `provider_vendors`. ## Common provider failure patterns ### JMBullion — `fractional_weight` - **Cause:** mega-menu lists fractional coins on every product page; pre-render HTML trips the fractional check before full render. - **Fix:** confirm `PROVIDER_CONFIG.jmbullion.fractionalExempt = true` and `phase: "cf-clearance-first"`. Verify Byparr sidecar is up. ### BullionExchanges — `page loaded, no price` - **Cause:** React pricing grid needs ~12–15s to hydrate; CF block redirects Firecrawl to homepage. - **Fix:** `phase: "cf-clearance-first"`, `waitFor: 15_000`, `timeout: 70_000`, `fractionalExempt: true`. Confirm Byparr `cf_clearance` cookie is fresh. ### Monument Metals — `out_of_stock — PRE-ORDER` - **Cause:** Legitimate pre-order (common Jan–Mar for new-year mintings). - **Fix:** Swap URL to an in-stock coin via the dashboard, or accept until inventory arrives. `monumentmetals` is in `PREORDER_TOLERANT_PROVIDERS`. ### APMEX / Hero / Gainesville — pipe tables - **Cause:** Pipe-delimited HTML tables — Playwright-direct `innerText` strips the pipes and `firstTableRowFirstPrice()` returns null, so `firstInRangePriceProse()` grabs a bulk "As Low As" price. - **Fix:** `phase: "firecrawl"` (already set) — Firecrawl markdown preserves the `|` pipes. Don't regress to `phase0`. ### SDBullion — wrong price from carousel - **Cause:** `sdbullion` was previously in `USES_AS_LOW_AS`; "Add on Items" carousel bled through. - **Applied fix (2026-02-20):** Removed from `USES_AS_LOW_AS`; added `MARKDOWN_CUTOFF_PATTERNS.sdbullion`. ## Firecrawl engine quick reference Self-hosted Firecrawl lives at `http://firecrawl-api:3002` on home poller and `http://localhost:3002` on Fly. Capabilities vary by image version — `waitFor` support in particular has changed across upgrades. Verify on the deployed version: ```bash # Home docker exec staktrakr-home-poller curl -s http://firecrawl-api:3002/health docker exec firecrawl-api node -e "console.log(require('/opt/firecrawl/package.json').version)" # Fly fly ssh console -a staktrakr -C "node -e \"console.log(require('/opt/firecrawl/package.json').version)\"" ``` If `waitFor` is silently ignored on the current image, PROVIDER_CONFIG's `waitFor` values are no-ops — use `phase: "cf-clearance-first"` or `phase0` (Playwright-direct) for JS-heavy SPAs instead of relying on Firecrawl to wait.