--- name: update-local-elections description: Refresh local-elections (общински избори) data — download ЦИК's csv.zip bundles + per-município HTML pages and rebuild data//. Use when the daily watch report flags "CIK local-elections results bundles" as changed, when the user asks to refresh local-elections data, ingest a new regular cycle (mi2027 etc.) or a partial-elections date (chmi*), or after a fresh git clone if data/2023_10_29_mi/ is missing. Bypasses Cloudflare via a headless Playwright session that warms a cf_clearance cookie once per run. allowed-tools: - Read - Bash - Edit - Write --- # Update local-elections skill Ingests Bulgarian local-elections data (общински съветници + кметове на община/кметство/район) from `results.cik.bg` into `data//`. Handles both regular cycles (`mi2019`, `mi2023`, future `mi2027`) and partial-elections dates under the rolling `chmi*` umbrella. Two historical archives are also supported as one-time backfills (not watched, since the data is frozen): `minr2015` (мест. избори + национален референдум, 2015-10-25) and `mipvr2011` (мест. избори + президентски избори, 2011-10-23). They use the same `--local-ingest ` CLI but require parser tolerance the modern cycles don't (different URL pattern `mestni/` vs `rezultati/`, missing CSS class markers, 3- and 4-column council tables, район mayor races split into separate `NNNN_NNNNNr.html` subpages on Sofia/Plovdiv/Varna). See "Historical cycles" below. ## When to run | Trigger | Action | |---|---| | Daily watcher reports `CIK local-elections results bundles: N cycle(s) changed: …` | Run incremental ingest for the changed cycle(s) — Step 2 below | | User asks to "refresh local elections" / "update мест. избори" / "ingest the new partial" | Same — Step 2 | | `data/2023_10_29_mi/` is empty on a fresh clone | Cold-start ingest with `--local-ingest mi2023` | | Adding a new regular cycle (e.g. mi2027 after Oct 2027) | First add the new slug → bundle URL to `scripts/watch/sources/cik_results.ts` `REGULAR_BUNDLE_URL`; then ingest | ## Architecture (read once) ``` results.cik.bg/{cycle}/csv.zip results.cik.bg/{cycle}/tur1/rezultati/{oikCode}.html results.cik.bg/{cycle}/tur2/rezultati/{oikCode}.html │ ▼ scripts/parsers_local/cik_fetch.ts ← Playwright warms cf_clearance once │ ▼ scripts/parsers_local/ingest_cycle.ts ← extract CP866-named zip, mirror HTML │ raw_data// ТУР1/{ОС,КО,КК,КР}/... ТУР2/{КО,КК,КР}/... html/tur1/*.html html/tur2/*.html │ ▼ scripts/parsers_local/parse_local_elections.ts │ data// index.json ← national rollup (council R1 votes by canonical party + mayors-won) municipalities/.json ← per-município bundle region/.json ← per-oblast rollup (region dashboard — one fetch; each município row carries electedMayor AND topCouncil to drive the council-support choropleth) regions_summary.json ← national per-oblast control (mayoral + council choropleths, top-regions); each row has topMayor + topCouncil national_leaders.json ← precomputed country-dashboard leaderboards (top mayors by %, closest races, split-control list, independent mayors) — one fetch instead of fanning out 265 município bundles index_trends.json ← trimmed sidecar (just councilVoteShare + mayorsByCanonical) so the cross-cycle trends tile pulls ~50 KB per cycle instead of ~100 KB _unmatched_coalitions.json ← operator-review queue ``` `region/.json` + `regions_summary.json` + `national_leaders.json` + `index_trends.json` are all produced by `scripts/parsers_local/build_region_json.ts` (`buildRegionRollups`), an additive bundle-only pass folded into the tail of `parseLocalElection` (and `resolveCanonicalsForCycle`). They regenerate automatically on every `--local-ingest` / `--local` / `--resolve-local-canonicals` run for regular `_mi` cycles, so no extra step is needed — chmi partials skip them (single-município, no region dimension). To rebuild only the rollups without re-fetching CIK HTML: `npm run data -- --local-rollups [--local-date ]`. Cross-cycle chmi history (`data/local_chmi_history.json`) is built by `scripts/parsers_local/build_chmi_history.ts` (`buildChmiHistory`) at the tail of `parseLocalElection` and `resolveCanonicalsForCycle`. The same builder additionally writes per-município shards to `data/chmi_history/.json` (rewritten from scratch each run) so the município page + settlement dashboard fetch their own ≤ 1 KB file instead of pulling the 61 KB global. The global file is kept for the national `/local/chmi` feed which needs every event. The fingerprint source `cik_results` in `scripts/watch/sources/cik_results.ts` tracks two kinds of cycle daily and queues this skill via `process-watch-report` when either changes: - **Regular cycles** (`mi2019`, `mi2023`) — HEADs the section bundle (`REGULAR_BUNDLE_URL`; mi2023's is `tur1/opendata/export.zip`, NOT a uniform `csv.zip`) for `Last-Modified` + `Content-Length`. - **Partial cycles** — enumerates every dated subdirectory under each `PARTIAL_UMBRELLAS` entry from CIK's root index, matching **both** `_chastichen` (partial) **and** `_nov` (new election, incl. full council re-elections). These are HTML-only; a newly-appeared date folder is the change signal. ## Step 1 — Prerequisites The parser cross-references canonical parties to credit local coalitions back to their primary national party. If `data/canonical_parties.json` is missing or stale: ```bash npm run data -- --summary ``` This regenerates the canonical parties index (~5s). ## Step 2 — Ingest the changed cycle(s) Read `state/watch/cik_results.json` to identify which cycles changed. The describe-line on the watcher report names them — e.g. `"2 cycle(s) changed: mi2023 re-uploaded, chmi2024-2026/2026-03-22_chastichen re-uploaded"`. For each changed cycle slug, run: ```bash npm run data -- --local-ingest ``` `--local-ingest` is the **HTML-only** path: it discovers the OIK catalogue from the cycle's dropdowns, mirrors the per-município pages, and parses. It does NOT download the section CSV bundle (that's `--local-csv`, see below). Examples: ```bash # A regular cycle was re-uploaded (HTML re-parse): npm run data -- --local-ingest mi2023 # A new partial-elections date appeared (either kind works the same): npm run data -- --local-ingest "chmi2024-2026/2026-03-22_chastichen" npm run data -- --local-ingest "chmi2024-2026/2025-10-12_nov" ``` Expected output: ``` [ingest_cycle] chmi2024-2026/…_chastichen :: discovering OIK catalogue [ingest_cycle] chmi2024-2026/…_chastichen :: discovered 10 OIK code(s) [ingest_cycle] chmi2024-2026/…_chastichen :: tur1=10, tur2=10, missing tur1=0 [parsers_local] 2025_10_12_chmi: wrote 10 município bundles to …/data/2025_10_12_chmi ``` **chmi structural variants are auto-handled** (no per-cycle flags): - Both `_chastichen` and `_nov` use the same id'd `obl-select`/`obs-select` dropdowns as mi2019/mi2023 (discovery is keyed off `usesIdSelects` in `ingest_cycle.ts`, which now includes any `chmi*` slug). - The single-round "нови избори за общински съветници" partials publish at `/rezultati/...` with **no `tur1/` segment** — `ingestCycle` probes the tur1 index, detects the 404, and switches to the roundless layout. A full council re-election surfaces in `/local/chmi` as a `council`-kind event (leading party + seats; see `build_chmi_history.ts`). The HTML mirror is sequential with a 250ms delay between requests — full mi2023 takes ~3 min. Partial ingests typically finish in under 30 s. ### Section-level CSV (votes + turnout per polling station) For **regular cycles only** (2015/2019/2023), `--local-csv ` additionally downloads the section bundle and re-parses, layering per-station data on top of the HTML bundles: ```bash npm run data -- --local-csv mi2023 # or minr2015 / mi2019 ``` This fixes the council vote share for cycles whose HTML summary omits it (2015), fills real `protocol` turnout, and writes the two-tier section data — `data//sections/.json` (light index: the map + top-sections + `LocalSectionsTile` table) + `data//sections//.json` (per-station full breakdown for the detail page). Follow with `--local-coords` (Step 5.5) to stamp GPS/address. Per-cycle bundle URLs + the cp866 extractor live in `download_csv_bundle.ts` / `extract_bundle.ts`. chmi partials are HTML-only — they have no section bundle, so `--local-csv` does not apply to them. **Cloudflare cookie**: the first request of the run launches headless Chromium to solve the JS challenge, captures `cf_clearance`, and persists it to `state/cik_clearance.json`. Subsequent requests in the same run use plain `fetch` with that cookie. The cookie typically lives ~30 min — long enough for one cycle's ~530 HTTP requests. If you see "Cloudflare challenge did not clear within 30 s — manual intervention required (cik_fetch)", CF has updated its challenge. Investigate `scripts/parsers_local/cik_fetch.ts` and consider raising the warm-up wait or adding a stealth plugin. ## Step 3 — Curate unmatched coalitions After ingest, inspect: ```bash cat data//_unmatched_coalitions.json ``` If the file has entries, each key is a raw local_party_name from `local_parties.txt` whose fragment lookup against `data/canonical_parties.json` failed. Add overrides to `scripts/parsers_local/local_coalition_overrides.ts`: ```ts // In localCoalitionRawOverrides { rawName: "Местна коалиция Граждани за X (ВМРО-БНД, БДЦ)", primaryCanonicalId: "vmro", memberCanonicalIds: ["vmro", "bdc"], }, // Or, if a fragment recurs across many local coalitions, in // localCoalitionFragmentOverrides: { fragment: "ВМРО-БНД", canonicalId: "vmro" }, ``` Re-run the parser only (skip the download/extract steps): ```bash npm run data -- --local --local-date ``` …until `_unmatched_coalitions.json` is empty or only contains genuinely-local entities you accept bucketing as `independent`. ## Step 4 — Verify ```bash node -e " const idx = require('./data/2023_10_29_mi/index.json'); console.log('cycle:', idx.cycle); console.log('municipalities:', idx.municipalities.length); console.log('top 5 council vote share:'); idx.councilVoteShare.slice(0, 5).forEach(p => console.log(' ', p.displayName.padEnd(30), p.pctOfValid.toFixed(2) + '%')); console.log('top 5 mayors won:'); idx.mayorsByCanonical.slice(0, 5).forEach(p => console.log(' ', p.displayName.padEnd(30), p.count)); " ``` For mi2023 expect roughly: - 265 municipalities - Top council vote share dominated by ГЕРБ, БСП, ДПС - Mayors-won dominated by ГЕРБ + independents (Blagoevgrad's Илко Стоянов won as an independent) Visual check: `npm run dev` → visit `/settlement/BLG03` → the new "Местни избори · 29.10.2023" tile should show Илко Стоянов Стоянов as elected mayor with R2 result. ## Step 5 — Post-ingest decorators After the parser writes (or rewrites) the per-município bundles, run the two decorators that enrich the candidate rows. Both walk every `data//municipalities/*.json` shard, so they cost ~10 s combined and are idempotent — re-running them when nothing has changed is a no-op. ```bash # Refresh kметство → EKATTE lookup the My-Area village-mayor tile reads. npx tsx scripts/parsers_local/backfill_kmetstvo_ekatte.ts # Stamp mpId on every mayor / councillor / kmetstvo / район candidate whose # normalised name matches an MP in data/parliament/index.json. Drives the # parliament.bg photo reuse in `MpAvatar`. Re-run automatically whenever # either local-election bundles OR parliament index changes — see # parliament-scrape SKILL.md for the inverse trigger. npx tsx scripts/parsers_local/decorate_local_mp_links.ts ``` The decorator's summary line reports `N/M candidate rows stamped (X%)` per cycle. National baseline is ~11% — most village councillors never sat in parliament. A sudden drop usually means the parliament index lost its `normalizedName` field; rerun `parliament-scrape` first. ## Step 5.5 — Section coordinates (powers the per-município section map) The local section shards carry no GPS of their own — stamp coordinates + building address onto the section index from the parliamentary section archive: ```bash npm run data -- --local-coords ``` Idempotent; walks every cycle. The join is settlement-name-gated (local and parliamentary section codes/numbers diverge — see `backfill_local_section_coords.ts`); ~98% coverage. Run after any section data lands, **and** after a new parliamentary election adds fresh section coordinates. Note the section data is now two-tier — every ingest/parse path emits a light `sections/.json` index (top-5 parties/station; drives the map + top-sections + table) plus per-station `sections//.json` full-breakdown files (the detail page fetches just one) via `emitSectionFiles`. ## Step 5.6 — Problem sections (powers the município "Risk votes" tile) Flag the curated Roma-neighborhood polling sections inside the local council data — the council-ballot analogue of the parliamentary `problem_sections` report. Writes `data//problem_sections.json` for every regular `_mi` cycle (the município dashboard's **Risk votes / Problem votes by party** tile reads it; self-hides for municípios with no flagged neighborhood): ```bash npm run data -- --local-problem-sections ``` **Run this AFTER `--local-coords`** — the match keys on EKATTE + the building `address` that the coords step stamps onto the shards. Reuses the watchlist in `scripts/reports/problem_sections/neighborhoods.ts` (same source as parliamentary); matching is local-specific (`problem_sections_local.ts`) because local section codes are NSI-oblast-prefixed while parliamentary are МИР-prefixed, so the codes can't be joined — it matches by section prefix (Plovdiv only, where МИР=NSI), else address keyword + normalized EKATTE. Reads the per-station detail files for full party votes. Idempotent; also folded into `--all`. ## Step 6 — Stamp the ingest marker `ingestCycles` writes `state/ingest/cik_local.json` automatically on success. If you ran the parser-only path (`--local --local-date …` rather than `--local-ingest …`), you can stamp manually: ```bash npx tsx scripts/stamp-ingest.ts cik_local --summary "manual parse of " ``` ## Adding a new regular cycle When CIK publishes mi2027 (~Oct 2027): 1. Edit `scripts/watch/sources/cik_results.ts` — add the slug → section-bundle URL (check CIK's actual path; it's moved each cycle: mi2019 `csv.zip`, mi2023 `tur1/opendata/export.zip`): ```ts const REGULAR_BUNDLE_URL: Record = { mi2019: `${ROOT}/mi2019/csv.zip`, mi2023: `${ROOT}/mi2023/tur1/opendata/export.zip`, mi2027: `${ROOT}/mi2027/…`, }; ``` 2. Edit `scripts/parsers_local/ingest_cycle.ts` `REGULAR_DATES`: ```ts const REGULAR_DATES: Record = { mi2019: "2019_10_27", mi2023: "2023_10_29", mi2027: "2027_10_xx", // ← actual election day }; ``` 3. Add a single-entry catalogue update to `src/data/json/local_elections.json`. 4. Ingest: `npm run data -- --local-ingest mi2027`. ## Historical cycles (mi2015 / mipvr2011) The 2015 (`minr2015`) and 2011 (`mipvr2011`) archives are usable via the same CLI: ```bash npm run data -- --local-ingest minr2015 npm run data -- --local-ingest mipvr2011 ``` They are **not watched** — the fingerprint source intentionally only tracks `mi2019` / `mi2023` / chmi umbrellas because the older archives are frozen, and adding them to `REGULAR_BUNDLE_URL` in `scripts/watch/sources/cik_results.ts` would just add HEAD requests for files that never change. Re-run the backfill manually if `data/2015_10_25_mi/` or `data/2011_10_23_mi/` is missing on a clone. Both **2015** (`--local-csv minr2015`) and **2011** (`--local-csv mipvr2011`) have section CSVs that backfill council vote share + turnout + per-station data. 2011's bundle (`el2011_t1.zip`) is a separate schema (CP1251 content, `общински съветници` folder, `coalitions` file, pair-encoded votes, no admin/serials columns) read by `augment_sections_2011.ts`, which the orchestrator falls back to automatically. NB: 2011's HTML council table was incomplete (omitted also-ran parties), so the CSV raises its council total from ~2.73 M to the complete ~3.27 M. What's different from 2019+ — already handled in the parser, listed here so you can debug a re-run intelligently: - **URL pattern**: pages live under `mestni/` not `rezultati/`. `RESULTS_PATH` in `ingest_cycle.ts` maps per-slug. - **OIK discovery**: 2015's index dropdown is unnamed (not `#obl-select`); 2011's lists 28 oblast codes as bare 2-digit values (`01`–`28`) with a JS-constructed redirect. Both fall through to `readLocationSelectOptions` + `scrapeOikRefs`, which find the largest redirecting `