---
name: atlas-tx-public-data-lanes
description: Build Atlas TX county/data feature lanes using structured public endpoints, county-normalized summaries, and honest map layers before heavier geospatial analytics.
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [atlas-tx, texas, public-data, socrata, arcgis, county-explorer, water]
    related_skills: [writing-plans, test-driven-development, polite-web-scraping]
---

# Atlas TX Public Data Lanes

Use when adding a new Atlas TX dataset lane (water, flood, social, fiscal, etc.) that must join cleanly to Texas counties and show up in APIs/UI fast.

## Core approach

1. Prefer machine-readable public endpoints over HTML scraping.
   - First choices: Socrata JSON, ArcGIS REST, USGS/NWS APIs.
   - Treat TWDB/TCEQ hub pages and similar portals as discovery sources first, not primary ingestion targets.

2. Build county summaries before heavy map analytics.
   - Get statewide county rollups working first.
   - Expose county detail + statewide overview APIs.
   - Render honest map layers first (county centroid dot map or point layers), not fake polygons.

3. Normalize county joins aggressively.
   - Reuse `normalizeCountyName()` and `countySlug()`.
   - Support slug inputs like `travis-county` by normalizing hyphens to spaces before title-casing.
   - Do not silently drop unmatched records; keep an unknown/unjoined bucket or note.

4. For Socrata sources, use grouped statewide queries first.
   - Extend query helpers to support `$select`, `$group`, `$order`, `$limit`, `$where`, `$having`.
   - Build county overviews from grouped statewide results, then use county-specific source collectors for detail pages.

5. For spatially heavy sources like FEMA NFHL:
   - Phase 1: metadata + query builder only.
   - Phase 2: county join strategy (county field, point-in-county, or polygon overlay).
   - Do not start with full statewide polygon analytics.

## Proven Atlas TX pattern

### Existing file patterns
- `src/lib/texas-open-data.ts` — grouped Socrata query builder + fetch helpers
- `src/lib/county-data-service.ts` — county detail collector pattern
- `src/lib/atlas-county-explorer.ts` — statewide overview + county breakdown orchestration
- `src/lib/texas-county-centroids.ts` — honest first-pass county map coordinates

### API pattern
Ship two endpoints first:
- `/api/<lane>/overview`
- `/api/<lane>/counties/[slug]`

Overview returns statewide county summaries for map/ranking.
County route returns selected county detail plus source slices/layers.

### UI pattern
- Overview map first
- Ranking table second
- Selected county breakdown cards third

## Implementation order for a new lane

1. Define normalized types in `src/lib/<lane>/types.ts`
2. Add source registry in `src/lib/<lane>/<lane>-source-registry.ts`
3. Add 2-5 adapters for the strongest verified sources
4. Build summary service that merges sources into county summaries
5. Add `/overview` and `/counties/[slug]` routes
6. Add tests before implementation for grouped queries, service merge logic, and routes
7. Add UI after APIs are green

## Water/flood-specific findings

Verified sources that are suitable starting points:
- FEMA NFHL ArcGIS REST: `https://hazards.fema.gov/arcgis/rest/services/public/NFHL/MapServer?f=pjson`
- USGS Texas stream sites: `https://waterservices.usgs.gov/nwis/site/?format=rdb&stateCd=tx&siteType=ST&siteStatus=active`
- NWS active alerts TX: `https://api.weather.gov/alerts/active?area=TX`
- TCEQ sanitary sewer overflows: `https://data.texas.gov/resource/8kc5-95uk.json?$limit=1`
- TCEQ general water permits: `https://data.texas.gov/resource/6pm5-am5m.json?$limit=1`
- Texas water districts: `https://data.texas.gov/resource/hr84-s96f.json?$limit=1`
- TCEQ Water Quality Individual Permits Active/Pending: `https://data.texas.gov/resource/7fq8-wig2.json?$limit=1`

### Texas city open-data discovery lesson

When the user wants municipal open-data discovery for Texas cities, do not assume one portal type.

Current verified split:
- Austin — Socrata (`https://data.austintexas.gov/api/search/views.json?limit=1` works)
- Dallas — Socrata (`https://www.dallasopendata.com/api/search/views.json?q=` works)
- Houston — CKAN (`https://data.houstontx.gov/api/3/action/package_search?rows=1` works)
- San Antonio — CKAN (`https://data.sanantonio.gov/api/3/action/package_search?rows=1` works)

Practical approach:
1. Probe the homepage first to identify the portal family (`Powered by Socrata`, `ckan`, etc.).
2. For Socrata portals, ingest catalog metadata from `/api/search/views.json` and normalize fields from `results[].view`.
3. For CKAN portals, ingest catalog metadata from `/api/3/action/package_search` and normalize from `result.results[]`.
4. Commit one statewide Texas-city snapshot if the file stays under ~5 MB compressed/minified enough for the repo budget.
5. Treat this as catalog metadata only, not row-level dataset ingestion.

Important caveats:
- Socrata catalog counts can include non-tabular assets (views, measures, files, filters), so downstream consumers may need to filter by `assetType`.
- CKAN timestamps may be timezone-qualified and normalize to UTC; expect apparent hour shifts in tests if you compare raw strings.
- Hosted meta-layers like tryopendata.ai may expose docs/UI but still block unauthenticated API probing; fall back to the city portals directly when that happens.

### City open-data curation lesson

After pulling the full portal catalogs, create a second curated snapshot for Atlas TX triage instead of jumping straight into row-level ingestion.

Recommended pattern:
1. Keep the raw all-catalog snapshot committed as `public/cache/city-open-data-tx.json`.
2. Build a second committed snapshot like `public/cache/city-open-data-curated-tx.json`.
3. Curate by keyword/theme matching across at least:
   - `name`
   - `description`
   - `category`
   - `organization`
4. Tag each curated row with:
   - `matchedThemes` (for example `water`, `permits`, `infrastructure`)
   - `matchReasons` as `field:keyword` strings for auditability
5. Add a separate refresh script (`refresh:city-open-data-curated`) so the curation step is reproducible and can evolve independently from raw catalog ingest.

Use this as a triage layer, not a truth layer:
- it is meant to shrink thousands of portal entries into a shortlist for follow-on ingestion;
- expect noisy matches, especially from generic words like `utilities`, `repair`, or `permit`;
- tests should assert theme presence and representative reasons, not brittle exact full reason arrays unless the classifier is intentionally fixed.

### Permit-lane lesson

For a stable Texas-wide pending-permits page, prefer the Socrata permit inventory first:
- dataset: `7fq8-wig2`
- useful fields: `permit_number`, `authorization_type`, `authorization_status`, `permittee_name`, `facility_county`, `nearest_city`, `latitude`, `longitude`
- statewide status counts can be pulled with grouped Socrata queries (`$select=authorization_status, count(*) as c` + `$group=authorization_status`)
- the pending-only roster can be pulled with `$where=authorization_status='PENDING'`

Do **not** start the first pending-permits UI from CID Search One if the goal is a reliable statewide tracker. CID is better treated as a secondary procedural lane because Search One is operationally fragile here. Use CID later for docket/protest overlays and caveat it clearly.

### City open-data ranking lesson

After the curated snapshot exists, add a third ranked snapshot instead of hand-picking rows ad hoc.

### Permit county-map backing lesson

When the user asks to "back" a county visualization before true polygon geometry exists, use the existing Texas county centroids to ship an honest interim map fast.

Recommended pattern for `/permits`-style pages:
1. Keep the page data loader simple (`getTceqPendingPermitsPageData`) and derive map rows separately.
2. Add a deterministic county aggregator in the data module that combines:
   - pending permit counts by county
   - CID open-case counts by county
   - hearing/public-meeting counts by county
3. Normalize county names with `normalizeCountyName()` and emit `slug` via `countySlug()` so the map joins directly against `TEXAS_COUNTY_CENTROIDS`.
4. In the page layer, project centroid lat/lon into a fixed SVG viewport with the same lightweight projection used on `/water`:
   - hard-coded Texas lat/lon bounds
   - `projectPoint()` + `clamp()` helpers
5. Render county-centroid tiles, not pretend polygons:
   - color band by permit count
   - hatch overlay when CID procedural pressure exists
   - include a title/tooltip string per county tile
6. Test both layers:
   - data test for county aggregation shape/order
   - page render test for presence of the map section and county tile markers (for example `data-county-map-tile="travis-county"`)

Important caveats:
- call it a projected county-centroid or county-tile map, not a full choropleth;
- use real county geometry later when available, but do not block shipping on that dependency;
- if build/test suddenly fail after rebasing because other work landed new dependencies, run `npm install` before treating it as a code bug.

Recommended pattern:
1. Keep three layers:
   - raw catalog: `public/cache/city-open-data-tx.json`
   - curated theme filter: `public/cache/city-open-data-curated-tx.json`
   - ranked ingest shortlist: `public/cache/city-open-data-ranked-tx.json`
2. Score curated rows into explicit Atlas lanes:
   - `water-utility-ops`
   - `flood-drainage`
   - `building-development-permits`
   - `sewer-wastewater`
   - `infrastructure-projects`
   - `deprioritized`
3. Use heuristic scoring from multiple signals, not theme labels alone:
   - positive signals: `water main`, `stormwater`, `drainage`, `outfall`, `permit`, `inspection`, `public works`, `capital project`
   - operational asset bonus for `assetType=dataset`
   - downrankers for noisy catalog artifacts like `measure`, `rating`, and `green building`
4. Persist per-row auditability:
   - `priorityLane`
   - `priorityScore`
   - `priorityReasons`
5. Emit a shortlist in the ranked snapshot (for example `priorityTop25`) so downstream Atlas planning can consume the queue directly.

Important findings:
- broad curated keywords massively over-match utility-adjacent Austin Energy content; ranking is the cleanup layer.
- theme-filtered rows should still be retained in the ranked snapshot even when deprioritized, so analysts can inspect what got pushed down.
- tests for ranking should assert lane assignment, relative ordering, and shortlist membership — not brittle exact top-1 expectations when several rows can land on similar scores.

Discovery-only initially:
- TWDB flood planning page
- TWDB GIS data page
- TCEQ GIS hub
- National Levee Database

## Scrape-planner lane (schema/hierarchy first)

When adding agentic scraping workflows, lock structure before execution concurrency.

Recommended reusable pattern (implemented in Atlas TX as `buildAtlasParallelScrapePlan`):
1. Build a source hierarchy first (`critical` / `supporting` / `context`) from dataset category + reliability.
2. Define one canonical row schema before any scrape run budgeting.
3. Only then compute per-source parallelism and request budgets from:
   - aggressiveness mode (`conservative` / `balanced` / `aggressive`)
   - source reliability score (0..1)
   - collection type penalty (`html-scrape` lower than structured APIs)
   - tier weight (critical > supporting > context)
4. Emit both global and per-source execution caps:
   - `maxGlobalParallelRequests`
   - `targetRequestsPerMinute`
   - `maxParallelRequests`

Why this matters:
- avoids over-optimizing concurrency before data contracts exist;
- makes scrape behavior tunable without changing ingest schema;
- de-risks fragile sources (especially browser/HTML lanes) by design.

Test expectations:
- verify hierarchy ordering and canonical schema presence independently of aggressiveness;
- verify aggressive mode increases total capacity versus conservative;
- verify lower-reliability HTML sources remain throttled relative to reliable API sources.

## TDD notes

For each new source:
1. Write a failing parser/normalizer test first.
2. Verify RED with the specific test file.
3. Implement the narrowest adapter possible.
4. Verify GREEN.
5. Run full suite.

Good first tests:
- grouped statewide query URL generation
- source parser normalization
- county summary merge logic
- route JSON shape

## Pitfalls hit and fixes

- Socrata grouped query URLs: URL encoding/order mattered in tests. Build query strings deterministically instead of relying on `URLSearchParams` ordering/encoding defaults.
- County slugs as inputs: `normalizeCountyName("travis-county")` originally failed; normalize hyphens to spaces before suffix handling.
- Local Next dev verification: API routes may work while homepage fails due to county slug normalization issues. Check both `/api/...` and `/` when validating.
- Browser automation against local Next dev can show HMR cross-origin warnings from `127.0.0.1`; prefer direct HTTP checks plus server logs for debugging local app issues.

## Rule of thumb

If a source can power a county summary without geometry-heavy joins, ship it now.
If it needs polygon math or brittle scraping, register it, document it, and defer it to phase 2.