--- name: atlas-tx-water-data-lane description: Build Atlas TX water/flood intelligence slices using verified public endpoints, county joins, and TDD-first adapters. version: 1.0.0 author: Hermes Agent license: MIT metadata: hermes: tags: [atlas-tx, water, flood, texas, nextjs, socrata, usgs, nws, tdd] related_skills: [writing-plans, test-driven-development] --- # Atlas TX Water Data Lane Use when extending `/home/stathis/Projects/atlas-tx` with water/flood features. ## What worked - Start with structured endpoints, not HTML scraping. - Prefer this order: 1. NWS alerts 2. USGS gauges 3. TCEQ sewer overflows 4. TCEQ general permits 5. governance/utility layers 6. FEMA polygon-heavy work later - Use strict TDD. Write failing tests for registry, adapter normalization, summary service, and API routes before implementation. ## Verified sources - FEMA NFHL ArcGIS REST: `https://hazards.fema.gov/arcgis/rest/services/public/NFHL/MapServer?f=pjson` - USGS Texas stream sites: `https://waterservices.usgs.gov/nwis/site/?format=rdb&stateCd=tx&siteType=ST&siteStatus=active` - NWS alerts for Texas: `https://api.weather.gov/alerts/active?area=TX` - TCEQ sewer overflows: `https://data.texas.gov/resource/8kc5-95uk.json?$limit=1` - TCEQ general water permits: `https://data.texas.gov/resource/6pm5-am5m.json?$limit=1` - Texas water districts: `https://data.texas.gov/resource/hr84-s96f.json?$limit=1` - PUCT investor-owned utilities: `https://data.texas.gov/resource/auk8-env9.json?$limit=1` - PUCT submeter utilities: `https://data.texas.gov/resource/iuez-sv34.json?$limit=1` Important: `auk8-env9` and `iuez-sv34` are live Socrata resources but are not in the current MVP dataset registry, so `fetchDatasetRows()` from `src/lib/texas-open-data.ts` will reject them. For governance work, fetch them directly from `/resource/.json` instead of routing through the MVP registry helper. ## File pattern Create under `src/lib/water/`: - `types.ts` - `water-source-registry.ts` - `county-lookup.ts` - `nws.ts` - `usgs.ts` - `tceq-sewer-overflows.ts` - `tceq-general-permits.ts` - `water-governance.ts` - `water-summary-service.ts` API routes: - `src/app/api/water/overview/route.ts` - `src/app/api/water/counties/[slug]/route.ts` - `src/app/api/water/alerts/route.ts` - `src/app/api/water/gauges/route.ts` UI: - `src/app/water/page.tsx` Tests: - `tests/water-source-registry.test.ts` - `tests/nws-water-alerts.test.ts` - `tests/usgs-water.test.ts` - `tests/tceq-sewer-overflows.test.ts` - `tests/tceq-general-permits.test.ts` - `tests/water-governance.test.ts` - `tests/water-governance-summary.test.ts` - `tests/water-summary-service.test.ts` - `tests/water-api-routes.test.ts` - `tests/water-alerts-route.test.ts` - `tests/water-gauges-route.test.ts` - `tests/water-page.test.ts` - `tests/water-page-governance.test.ts` ## County join approach - Reuse `normalizeCountyName()` and `countySlug()`. - Add a `county-lookup.ts` helper backed by `src/lib/texas-county-centroids.ts` so county FIPS can resolve to county names/slugs. - Important bug found: slugs like `travis-county` must normalize back to `Travis County`. Fix by replacing hyphens with spaces inside `normalizeBaseCountyName()` in `src/lib/counties.ts`. ## Adapter specifics ### NWS - Normalize `areaDesc` by splitting on commas/semicolons. - Convert county tokens into `Travis County` form. - Filter to water-relevant events only (Flood Warning, Flash Flood Warning, Flood Advisory, Hydrologic Outlook, Coastal Flood Warning, Flood Watch, Flash Flood Statement). ### USGS - NWIS site service is tab-delimited RDB, not JSON. - Skip `#` comment lines. - First non-comment row = headers, second = schema hints, remaining = data. - Texas county FIPS from NWIS are 3-digit county codes; prefix with `48`. - Resolve county name from FIPS using `county-lookup.ts`. ### TCEQ sewer overflows - Normalize `incident_location_county` through county helpers. - Convert `amount_unit` to gallons. - Explicitly handle `MGD` as `value * 1_000_000`. - Summaries should return both count and gallons by county slug. ### TCEQ general permits - Add `tceq-general-permits.ts` once the first water slice is green. - Normalize `county_name` through county helpers. - Preserve `permit_no`, `permit_status`, `permit_type`, `site_name`, and lat/lon. - County summaries should at least expose `generalPermitCount` before doing deeper permit-status analytics. ### Governance layers - Add `water-governance.ts` after permits are stable. - Normalize TCEQ water districts from `hr84-s96f` into `WaterGovernanceEntity` with: - `district_number` -> `entityId` - `distict_name` -> `entityName` - `district_type` -> `entityType` - `activity_status` -> `activityStatus` - Normalize PUCT utilities from direct Socrata `/resource/*.json` calls, not the MVP registry helper. - For `puct-water-iou`, set `entityType` to `Investor-Owned Utility`. - For `puct-water-submeter`, preserve `utility_type` when present and fall back to `Submeter Utility`. - Use `primary_county` for the first county join; do not explode `all_counties` yet. - Summary counts should split: - `waterDistrictCount` for `tceq-water-districts` - `waterUtilityCount` for all PUCT utility records ## Summary service - Aggregate alerts, gauges, sewer overflows, then general permits, then governance entities into `CountyWaterSummary`. - Build overview first, then county breakdown from the same raw collections. - Initial overlays should expose booleans for gauge/alert/sewer layers; leave floodplain false until FEMA work lands. - Add `generalPermitCount` before attempting a richer permit score. - Governance should add `waterDistrictCount` and `waterUtilityCount`. - County breakdown layers should include `permits` and `governance` alongside alerts/gauges/sewer overflows. - For utility governance, count `tceq-water-districts` separately from PUCT utility sources. - After FEMA county join lands, wire `metrics.floodplainFeatureCount` from joined NFHL political-jurisdiction counts and set `overlays.hasFloodplainLayer` from that real count. - If a county only appears through FEMA coverage, use the joined county coverage record as a fallback source of truth so `getCountyWaterBreakdown()` still resolves the county and returns a note like `NFHL political jurisdictions mapped: N`. - Do not force a composite water score in the first slice. ## EPA + Tap Score expansion - Add EPA before Tap Score if you need a new serious data lane. - Best next EPA source is the ECHO SDWA national download, not ad-hoc page scraping: - index: `https://echo.epa.gov/tools/data-downloads` - ZIP: `https://echo.epa.gov/files/echodownloads/SDWA_latest_downloads.zip` - summary/dictionary: `https://echo.epa.gov/tools/data-downloads/sdwa-download-summary` - Verified SDWA package tables useful for Atlas Texas: - `SDWA_PUB_WATER_SYSTEMS.csv` - `SDWA_FACILITIES.csv` - `SDWA_GEOGRAPHIC_AREAS.csv` - `SDWA_EVENTS_MILESTONES.csv` - `SDWA_VIOLATIONS_ENFORCEMENT.csv` - Use SDWA for Texas public-water-system coverage, violations, enforcement, population served, and county joins. This should become the primary drinking-water compliance lane. - Add EPA WQI as the second EPA lane: - archive: `https://echo.epa.gov/files/echodownloads/Data-Analytics/WQI/` - homepage: `https://echo.epa.gov/maps/water-quality-indicators` - Verified WQI archive exposes reusable station/sample bundles such as: - `wqi_stations_summary_.zip` - `wqi_monthly_sample_data_.zip` - Use WQI for Texas ambient monitoring context via point-in-county joins; do not confuse it with utility compliance. - EPA PFAS is useful but phase it after SDWA/WQI: - `https://echo.epa.gov/tools/data-downloads/national-pfas-datasets` - Tap Score is useful as enrichment only, not as a backbone source of truth. - Verified public Tap Score assets: - research page: `https://mytapscore.com/pages/us-water-data` - algorithm page: `https://mytapscore.com/pages/tap-score-algorithm` - Texas city pages exist for at least Houston, Dallas, Austin, San Antonio, Fort Worth, El Paso, McAllen, Arlington, Corpus Christi, Plano, Lubbock, and Amarillo. - Important interpretation: Tap Score is sample-based and proprietary. It should not drive county truth or regulatory scoring. - Good Tap Score use in Atlas Texas: - city-level enrichment only - nearby utility/test counts and visible city-page contaminant snippets - methodology inspiration for future consumer-facing health/aesthetic/pipe framing - Bad Tap Score use: - primary county score backbone - compliance truth - wide scraping assumptions without explicit API/data-sharing permission - If researching these sources live, skip Google search in browser mode here; it commonly trips anti-bot blocks. Use direct known URLs, sitemaps, and provider index pages instead. ## LCRA lane - LCRA is worth adding as a basin-specific operational layer for Atlas Texas, not as a statewide backbone. - Prioritize LCRA in this order: 1. Hydromet stage/flow 2. Hydromet lake levels 3. ARRP wastewater outfalls 4. ARRP land application permits 5. LCRA water-quality site inventory 6. CRWN volunteer network later - Verified first implementation slice that shipped cleanly: - adapter file: `src/lib/water/lcra-hydromet.ts` - route files: - `src/app/api/water/lcra/hydromet/stage-flow/route.ts` - `src/app/api/water/lcra/hydromet/lake-levels/route.ts` - tests: - `tests/lcra-hydromet.test.ts` - `tests/lcra-hydromet-routes.test.ts` - Normalization shape that worked well: - stage/flow -> `LcraStageFlowReading` - `sourceId: 'lcra-hydromet-stageflow'` - `siteNumber` as string - `stationName` from `location` - `observedAt` from `dateTime` - `stageFeet`, `flowCfs`, `bankfullFeet`, `floodStageFeet` - lake levels -> `LcraLakeLevelReading` - `sourceId: 'lcra-hydromet-lakelevels'` - `siteNumber` as string - `stationName` from `location` - `observedAt` from `dateTime` - `elevationFeet` - Route behavior that worked: - both routes accept optional `?siteNumber=` and filter server-side - response shape should be: - `{ readings, freshness }` - use `buildWaterFreshness(['lcra-hydromet-stageflow'])` or `buildWaterFreshness(['lcra-hydromet-lakelevels'])` - Cache/freshness implementation detail worth keeping: - map source IDs in `src/lib/water/freshness.ts` - cache keys used: - `lcra-hydromet-stageflow` - `lcra-hydromet-lakelevels` - `lcra-arrp-outfalls` - `lcra-arrp-land-permits` - a 5-minute TTL worked for the first live Hydromet slice and matches operational expectations better than daily-style cadences - a 12-hour TTL worked for the first ARRP slice because those permit/outfall inventories are not ultra-realtime - Registry gotcha: - after adding new LCRA source IDs to `src/lib/water/water-source-registry.ts`, update `tests/water-source-registry.test.ts` immediately or the full suite will fail on expected source count/order and source count - Verified live LCRA endpoints: - Hydromet stage/flow: `https://hydromet.lcra.org/api/GetStageFlowForAllSites/` - Hydromet lake levels: `https://hydromet.lcra.org/api/GetLakeLevelsForAllSites` - Water-quality sites: `https://waterquality.lcra.org/api/Sites/GetSites` - ARRP outfalls: `https://waterquality.lcra.org/arrp/api/Outfall` - ARRP land permits: `https://waterquality.lcra.org/arrp/api/LandPermit` - ARRP slice that shipped cleanly: - adapter file: `src/lib/water/lcra-arrp.ts` - route files: - `src/app/api/water/lcra/arrp/outfalls/route.ts` - `src/app/api/water/lcra/arrp/land-permits/route.ts` - tests: - `tests/lcra-arrp.test.ts` - `tests/lcra-arrp-routes.test.ts` - Important ARRP normalization detail: - both ARRP endpoints return an ASP.NET-style task envelope, not a bare array - shape observed live: `{ result: [...], id, exception, status, isCompleted, ... }` - extract `payload.result` first, then normalize records - Normalization shape that worked for ARRP: - outfalls -> `LcraArrpOutfall` - `sourceId: 'lcra-arrp-outfalls'` - `recordId` from `objectid` - `permitNumber` from `permitNum` - `permitteeName` from `permittee` - `countyName` normalized via `normalizeCountyName()` - `status`, `segmentId`, `basinId`, `outfallNumber`, `latitude`, `longitude` - land permits -> `LcraArrpLandPermit` - `sourceId: 'lcra-arrp-land-permits'` - `recordId` from `recordId` - `permitNumber` from `permitNum` - `permitteeName` from `permittee` - `countyName` normalized via `normalizeCountyName()` - `status`, `segmentId`, `basinId`, `permitType` from `permittype`, `reviewType` from `revtype`, `latitude`, `longitude` - Route behavior that worked for ARRP: - `/api/water/lcra/arrp/outfalls?county=` returns `{ outfalls, freshness }` - `/api/water/lcra/arrp/land-permits?county=` returns `{ permits, freshness }` - county filtering is server-side via slug comparison using `countySlug()` - Verified live shape of the LCRA data during research: - Hydromet stage/flow returned 68 rows - Hydromet lake levels returned 24 rows - Water-quality sites returned 654 rows, including 343 active rows, 194 `Agency='LCRA'` rows, and 169 `ImpairedSegment=true` rows - ARRP outfalls returned 156 rows, with 147 `Current` and 9 `Pending` - ARRP land permits returned 204 rows, with 195 `Current` and 9 `Pending` - Hydromet useful fields: - `siteNumber`, `location`, `dateTime`, `stage`, `flow`, `bankfull`, `floodStage`, `elevation` - Water-quality site useful fields: - `SiteId`, `SiteName`, `SegmentId`, `SegmentName`, `RootSegmentId`, `Latitude`, `Longitude`, `LastDate`, `Agency`, `IsActive`, `ImpairedSegment` - ARRP useful fields: - `permitNum`, `permittee`, `status`, `county`, `latDd`, `longDd`, `segment`, `basin` - outfalls also expose `outfall` - Product fit: - use LCRA for Colorado-basin county intelligence, especially Travis/Burnet/Llano/Bastrop/Fayette/Colorado/Wharton/Matagorda and upper-basin segment-14 counties - do not present LCRA as a uniform statewide source - Modeling recommendation: - source IDs: `lcra-hydromet-stageflow`, `lcra-hydromet-lakelevels`, `lcra-water-quality-sites`, `lcra-arrp-outfalls`, `lcra-arrp-land-permits`, `lcra-crwn` - entity types: `hydromet_gauge`, `lake_level_station`, `basin_water_quality_site`, `wastewater_outfall`, `land_application_permit` - county metrics: `hydrometGaugeCount`, `activeFloodGaugeCount`, `reservoirStationCount`, `waterQualitySiteCount`, `activeWaterQualitySiteCount`, `impairedSegmentSiteCount`, `wastewaterOutfallCount`, `landApplicationPermitCount` - Important caveat: LCRA’s water-quality site portal explicitly points users to TCEQ for assessment/publication/regulatory use. Treat LCRA water-quality records as monitoring/discovery/context unless you later anchor them to TCEQ final datasets. - Discovery trick that worked: browser network/resource inspection exposed the hidden JSON endpoints faster than page scraping. For LCRA-style portals, inspect `performance.getEntriesByType('resource')` or page JS first, then hit discovered `/api/...` URLs directly. - Verified deeper LCRA quality endpoints behind `waterquality.lcra.org`: - site metadata: `https://waterquality.lcra.org/api/Sites/GetSite/` - site parameter catalog: `https://waterquality.lcra.org/api/Sites/GetSiteParameters/` - site observations: `https://waterquality.lcra.org/api/Sites/GetSiteData/` - site observations with profile rows: `https://waterquality.lcra.org/api/Sites/GetSiteData/?IncludeProfile=true` - segment metadata: `https://waterquality.lcra.org/api/Segments/GetSegment/` - segment parameter catalog: `https://waterquality.lcra.org/api/Segments/GetSegmentParameters/` - segment observations: `https://waterquality.lcra.org/api/Segments/GetSegmentData/` - Verified live shapes from those endpoints: - `GetSite` returns `SiteId`, `SiteName`, `SegmentId`, `SegmentName`, `RootSegmentId`, `RootSegmentName`, `Latitude`, `Longitude`, `LastDate`, `Agency`, `IsActive`, `ImpairedSegment`, `SurfaceDataOverride` - `GetSiteParameters` rows return `SiteId`, `SegmentId`, `StoretCode`, `StoretName`, `StoretCategory`, `HasSurfaceData` - `GetSiteData` rows return `SiteId`, `SegmentId`, `StoretCode`, `StoretName`, `StoretCategory`, `Depth`, `Agency`, `Symbol`, `Value`, `Date` - `GetSegment` returns `SegmentId`, `SegmentName`, `RootSegmentId`, `RootSegmentName`, `SiteIds`, `Agencies`, `ImpairedSegment`, `Sites[]` - First reusable adapter slice that shipped cleanly: - adapter file: `src/lib/water/lcra-water-quality.ts` - route files: - `src/app/api/water/lcra/quality/sites/route.ts` - `src/app/api/water/lcra/quality/sites/[siteId]/route.ts` - `src/app/api/water/lcra/quality/sites/[siteId]/parameters/route.ts` - `src/app/api/water/lcra/quality/sites/[siteId]/observations/route.ts` - tests: - `tests/lcra-water-quality.test.ts` - `tests/lcra-water-quality-routes.test.ts` - Normalization shape that worked: - `LcraWaterQualitySite` - `sourceId: 'lcra-water-quality-sites'` - `siteId`, `siteName`, `segmentId`, `segmentName`, `rootSegmentId`, `rootSegmentName`, `latitude`, `longitude`, `lastObservedAt`, `agency`, `isActive`, `impairedSegment`, `surfaceDataOverride` - `LcraWaterQualityParameter` - `sourceId: 'lcra-water-quality-parameters'` - `siteId`, `segmentId`, `storetCode`, `storetName`, `storetCategory`, `hasSurfaceData` - `LcraWaterQualityObservation` - `sourceId: 'lcra-water-quality-observations'` - `siteId`, `segmentId`, `storetCode`, `storetName`, `storetCategory`, `depth`, `agency`, `symbol`, `value`, `observedAt` - Route behavior that worked: - `/api/water/lcra/quality/sites` returns `{ sites, freshness }` - `/api/water/lcra/quality/sites/[siteId]` returns `{ site }` - `/api/water/lcra/quality/sites/[siteId]/parameters` returns `{ parameters }` - `/api/water/lcra/quality/sites/[siteId]/observations` returns `{ observations }` with optional `?storetCode=` and `?includeProfile=true` - Segment slice that shipped cleanly: - route files: - `src/app/api/water/lcra/quality/segments/[segmentId]/route.ts` - `src/app/api/water/lcra/quality/segments/[segmentId]/parameters/route.ts` - `src/app/api/water/lcra/quality/segments/[segmentId]/observations/route.ts` - route behavior: - `/api/water/lcra/quality/segments/[segmentId]` returns `{ segment }` - `/api/water/lcra/quality/segments/[segmentId]/parameters` returns `{ parameters }` - `/api/water/lcra/quality/segments/[segmentId]/observations` returns `{ observations }` with optional `?storetCode=` and `?includeProfile=true` - Segment normalization shape that worked: - `LcraWaterQualitySegment` - `sourceId: 'lcra-water-quality-segments'` - `segmentId`, `segmentName`, `rootSegmentId`, `rootSegmentName` - `siteIds` from both `SiteIds` CSV and `Sites[]` merged + deduped - `agencies` from CSV split - `impairedSegment`, `siteCount` - Important segment implementation pitfall: - `GetSegment` can expose both a `SiteIds` CSV string and a `Sites[]` array. - Do not choose one or the other; merge both sources and dedupe, or tests/live payloads will disagree on site coverage. - Caching detail: - cache key `lcra-water-quality-sites` should be wired into `src/lib/water/freshness.ts` - site metadata, parameter catalogs, and observation histories can use per-site keys like: - `lcra-water-quality-site:` - `lcra-water-quality-parameters:` - `lcra-water-quality-observations::surface|profile` - a 12-hour TTL worked fine for this first slice - Important environment quirk discovered during research: - `waterquality.lcra.org` TLS verification can fail in some local Python/Node contexts on this machine (`unable to verify the first certificate` / `UNABLE_TO_VERIFY_LEAF_SIGNATURE`) - browser access works; tests should stay mocked/offline - do not write live network tests against these endpoints in this environment - when wiring LCRA quality into the shared summary service, fail soft instead of letting these TLS errors break unrelated county summaries or page tests. A small helper like `safeLoad(() => fetcher(), fallback)` worked well for optional LCRA enrichment. - LCRA county aggregation slice that shipped cleanly: - county assignment: use site coordinates with a nearest-county fallback from `src/lib/texas-county-centroids.ts` - summary metrics that proved useful: - `activeLcraQualitySiteCount` - `latestLcraObservationAt` - `availableLcraParameterCount` - `impairedLcraMonitoringSiteCount` - county-detail notes that proved useful: - total / active / impaired site counts - latest observation timestamp - top STORET summary such as `DO (2), Temperature (1), Phosphorus (1)` - Important performance/scoping lesson for county aggregation: - do not fan out per-site parameter/observation fetches in the statewide overview path unless you really need them. - a better split is: - overview: derive lightweight counts from site inventory only (`IsActive`, `ImpairedSegment`, `LastDate`) - county detail: fan out parameter + observation fetches only for the selected county, then compute `availableLcraParameterCount` and top STORET summaries there. - this keeps the overview fast and avoids turning optional basin enrichment into a global latency bottleneck. ## Cache layer - Add a shared in-memory TTL cache at `src/lib/water/cache.ts` with `createWaterDataCache()` and `getGlobalWaterDataCache()`. - Use `getOrLoad(key, ttlMs, loader)` so concurrent callers share the same pending Promise and repeated calls inside the TTL do not re-fetch. - Good default TTLs from this implementation: - NWS alerts: 15 minutes - USGS gauges: 6 hours - TCEQ sewer overflows: 6 hours - TCEQ general permits: 12 hours - Water districts/utilities: 24 hours - FEMA metadata/political jurisdictions/levees: 24 hours - LCRA Hydromet: 15-60 minutes depending on UI expectations - LCRA ARRP + water-quality inventories: 12-24 hours - If a fetch function receives an explicit `AbortSignal`, bypass the cache and call the uncached loader directly. - Cache only broad statewide fetches. If a function accepts bounded params like `limit`, do not reuse the generic cached entry for those parameterized calls. - Extend the cache with `getFreshness(key)` returning `{ cached, cachedAt, expiresAt, ttlMs }` so API routes can expose cache state without re-fetching. - Add a `src/lib/water/freshness.ts` helper that maps public `sourceId`s to cache keys and builds a `{ generatedAt, sources }` freshness payload for API responses. - Good first freshness surfaces: `/api/water/overview`, `/api/water/alerts`, `/api/water/gauges`, and `/api/water/fema/nfhl/levees`. - After freshness lands, expose it in the UI as a dedicated `Source freshness` section on `/water` with one badge/card per source showing `Cached until ` and TTL in minutes. This makes live-demo reliability visible without opening APIs. - Once levee filtering works, add `/api/water/fema/nfhl/levees-by-county` that summarizes levees into `{ county, leveeCount, leveeNames, leveeIds }` and attach freshness metadata there too. ## Source provenance anomaly + dependency network lane - Water source provenance route can safely support month-level evidence drilldown without breaking existing consumers: - `GET /api/water/sources/[slug]?month=YYYY-MM` - return full profile plus `evidenceWindow: { month, timelinePoint, communitySamples }` - validate month format with `^\d{4}-\d{2}$`; return `400` for invalid input - In `/water/sources/[slug]`, anomaly cards should deep-link to evidence and exact timeline rows: - API link: `/api/water/sources/?month=` - row anchor id pattern: `timeline-` - First county dependency graph slice that shipped cleanly: - adapter/service: `src/lib/water/source-network.ts` - route: `src/app/api/water/sources/network/route.ts` - tests: - `tests/water-source-network.test.ts` - `tests/water-source-network-route.test.ts` - Reusable v1 dependency schema that worked: - `nodes[]`: `countySlug`, `countyName`, `lat`, `lon`, `multiCountySourceCount`, `contagionScore` - `edges[]`: undirected shared-provider links with `sharedSourceCount` + sample `sharedSources` - `directedEdges[]`: `upstreamCountySlug`, `downstreamCountySlug`, `sharedSourceCount`, `sharedSources` - `flowDirectionMethod`: explicitly label heuristic, e.g. `centroid-gulf-proxy-v1` - Hydrology-directed bootstrap lane that shipped cleanly: - seed file: `src/lib/water/hydrology-seed-edges.ts` - service: `src/lib/water/hydrology-dependencies.ts` - route: `src/app/api/water/sources/network/hydrology/route.ts` - tests: - `tests/water-hydrology-dependencies.test.ts` - `tests/water-hydrology-dependencies-route.test.ts` - stable method label: `seeded-river-network-v1` - node metrics that proved useful in UI: - `upstreamContributionScore` - `downstreamDependencyScore` - `contagionScore` - Network UI lane that shipped cleanly: - page route: `src/app/water/network/page.tsx` - nav entry from `/water` via `RoutePill` to `/water/network` - tests: - `tests/water-network-page.test.tsx` - `tests/water-network-scatter.test.tsx` - `tests/water-network-neighbors.test.tsx` - `tests/water-page-network-link.test.tsx` - high-signal UX pattern: 1. map + scatter on same page 2. URL-driven brush state: `?county=&scope=#network-workspace` 3. deep-link every node/point to that brush permalink 4. expose stable test hooks: `data-selected-county`, `data-selected-scatter`, `data-scatter-county` 5. add selected-county neighborhood panel with `Upstream counties` / `Downstream counties` lists and edge `evidence` 6. include direct drill links to `/water/counties/` and `/water/sources/` - Correlation analytics lane that shipped cleanly: - helper: `src/lib/water/network-analytics.ts` - first metrics: Pearson correlations for - upstream ↔ downstream - downstream ↔ contagion - upstream ↔ contagion - keep correlation cards deterministic and test-backed before adding more complex stats - Important honesty constraint: - shared-provider direction is NOT real hydrology; treat as dependency proxy only - always ship method metadata (`flowDirectionMethod`) so UI/API consumers know whether flow direction is inferred or physically modeled ## Hydrology dependency map + correlation lane - After shipping proxy `source-network` direction, add a separate hydrology-specific graph route and keep method labels explicit. - First clean slice that worked: - seed edge dataset: `src/lib/water/hydrology-seed-edges.ts` - graph service: `src/lib/water/hydrology-dependencies.ts` - API route: `src/app/api/water/sources/network/hydrology/route.ts` - page: `src/app/water/network/page.tsx` - tests: - `tests/water-hydrology-dependencies.test.ts` - `tests/water-hydrology-dependencies-route.test.ts` - `tests/water-network-page.test.tsx` - `tests/water-network-scatter.test.tsx` - `tests/water-network-analytics.test.ts` - Reusable schema that worked: - `flowDirectionMethod: 'seeded-river-network-v1'` - `nodes[]` with `upstreamContributionScore`, `downstreamDependencyScore`, `contagionScore`, and centroid fields - `edges[]` with directed `upstreamCountySlug`, `downstreamCountySlug`, `weight`, `evidence` - Scoring approach that produced useful ranking behavior: - `downstreamDependencyScore` = weighted incoming edges - `upstreamContributionScore` = weighted outgoing edges - `contagionScore` = blend of downstream-heavy + upstream contribution (first pass used 0.7 / 0.3) - UI pattern that shipped cleanly and is easy to test: 1. map panel with line segments + node circles (centroid projection) 2. ranked table for top downstream dependency counties 3. correlation KPI strip (`Upstream↔Downstream`, `Downstream↔Contagion`, `Upstream↔Contagion`) 4. scatter plot (`x=upstream`, `y=downstream`, `bubble size=contagion`) with URL-scoped mode toggle (`scope=all|top10`) - Testing/stability details worth preserving: - give server page components default props when adding optional `searchParams` (e.g. `function Page({searchParams} = {})`) so existing tests calling `pageModule.default()` do not crash - add stable DOM hooks for scatter points (`data-scatter-county=`) for resilient tests - keep route/page copy assertions focused on semantic labels (`Dependency scatter`, axis text, flow method) rather than brittle coordinate values - Product honesty rule: - seeded edges are a bootstrap, not full reach-level truth; keep `flowDirectionMethod` visible in UI copy and API payload until NHDPlus/USGS connectivity replaces seeds. ## Hydrology dependency map lane (seeded DAG bootstrap) - For "who depends on who downstream" UX, a practical first slice is a seeded directed county graph before full NHDPlus ingest. - Files that shipped cleanly: - `src/lib/water/hydrology-seed-edges.ts` - `src/lib/water/hydrology-dependencies.ts` - `src/app/api/water/sources/network/hydrology/route.ts` - `src/app/water/network/page.tsx` - Route contract that worked: - `GET /api/water/sources/network/hydrology` - payload: `{ generatedAt, schemaVersion, flowDirectionMethod, nodes, edges }` - `flowDirectionMethod` value: `seeded-river-network-v1` - Node scoring that worked for county ranking: - `upstreamContributionScore`: sum of outgoing edge weights - `downstreamDependencyScore`: sum of incoming edge weights - `contagionScore`: weighted blend (first pass used `0.7*downstream + 0.3*upstream`) - UI pattern that worked for fast demoability: - add `/water/network` as a dedicated map-first route - draw lines from `upstreamCountySlug -> downstreamCountySlug` - draw centroid circles sized by `contagionScore` - include a ranked table (`Top downstream dependency counties`) - add a shortcut pill from `/water` to `/water/network` - Test pattern that worked: - page test with mocked hydrology service (`renderToStaticMarkup`) asserting: - route title text - flow method label - county names - ranking section label - route test asserting API passthrough shape - Scope guardrail: - keep this lane explicitly labeled as seeded/proxy until NHDPlus/USGS reach connectivity is integrated. - do not imply regulatory or physically exact downstream routing while on seed edges. ## Route and UI lessons - `/api/water/alerts` and `/api/water/gauges` should accept optional `?county=` and filter server-side. - A first useful UI is `src/app/water/page.tsx` with a county table + selected-county detail panel; do this before maps if the API surface is still moving. - Add governance counts directly to the county table and county detail once the governance adapter lands; this gives useful value before map work. - After FEMA county coverage is real, add a simple SVG centroid map before chasing true county polygons. This is fast, demoable, and enough to visualize relative county risk. - For the centroid map, derive points from `src/lib/texas-county-centroids.ts` and project lat/lon into a fixed SVG box; style counties by a simple heuristic combining `floodplainFeatureCount`, alert count, sewer-overflow count, and gauge count. - Render selected county gauges as separate overlay markers and expose stable `data-county-slug` / `data-gauge-site` attributes so page tests can assert on map content without brittle pixel checks. - Add a direct link to `/api/water/fema/nfhl/counties` from the `/water` page once the FEMA county join route exists. - For server-component page tests, render the returned element with `renderToStaticMarkup()` instead of `JSON.stringify()` or `util.inspect()` if you need to assert on real text content. - Next.js lint catches `module` as a forbidden variable name in tests (`@next/next/no-assign-module-variable`). Use names like `pageModule` instead. - Avoid `any` in mocked route tests; use small inline structural types so `npm run lint` stays green. - For `/water/sources/[slug]` analytical UX, a high-signal pattern is: 1. compute monthly anomaly score from timeline rows with `openSignal = alerts + sewerOverflows + permits` 2. compute `coverageRatio = communitySampleCount / openSignal` (0 when openSignal is 0) 3. blend normalized pressure and coverage gap into a 0-100 score (e.g., pressure-heavy weighting) 4. bucket severity (`low`, `moderate`, `high`) and color timeline rows + badges by severity 5. add a ranked `Top anomalous months` panel with jump links into anchored timeline rows (`id=timeline-YYYY-MM`) and an evidence API link - Keep anomaly panel deterministic and testable: derive it entirely from `profile.timeline`, sort by score descending, slice top N, and assert rendered labels/links in `tests/water-source-page.test.tsx`. - When expanding `WaterBreakdown.layers`, update all test fixtures immediately or page tests will fail on missing properties like `governance.length`. - When `/water` starts reading `overview.freshness`, update all mocked `getWaterOverview()` fixtures in page tests to include at least `freshness: { generatedAt, sources: {} }`, otherwise server-component tests will crash before assertions. - When adding a new lane after another lane has already evolved `CountyWaterSummary` or `WaterBreakdown` (for example LCRA after surface-water-quality), re-read and merge the latest `src/lib/water/types.ts` and `src/lib/water/water-summary-service.ts` before editing. These files are active integration hubs and are likely to diverge on `metrics`, `overlays`, and `layers`. - Specifically, LCRA county-detail/UI work must preserve newer surface-water fields while adding: - metrics: `lcraArrpOutfallCount`, `lcraArrpLandPermitCount` - layers: `lcraArrpOutfalls`, `lcraArrpLandPermits` - notes: append LCRA ARRP counts without removing existing FEMA or surface-water notes - If a rebase conflict hits `types.ts` or `water-summary-service.ts`, do not pick one side. Manually merge both feature sets, then rerun the full verification loop before pushing. ## Verification Run: - `npm test` - `npm run lint` - `npm run build` Expected first useful routes: - `/api/water/overview` - `/api/water/counties/[slug]` ## FEMA discovery stub - Add `src/lib/water/fema-nfhl.ts` before any polygon-heavy ingest. - Keep the first FEMA slice to discovery only, then a thin live query route: - `fetchNfhlServiceMetadata()` - `normalizeNfhlServiceMetadata(...)` - `extractCountyRelevantNfhlLayers(...)` - `buildNfhlLayerQueryUrl(...)` - `fetchNfhlDiscoveryBundle()` - `buildTexasPoliticalJurisdictionsQueryUrl(limit?)` - `normalizeNfhlPoliticalJurisdictionsResponse(...)` - `fetchTexasNfhlPoliticalJurisdictions(limit?)` - Verified current county-relevant discovery targets from live metadata: - `Political Jurisdictions` (layer 22) - `Levees` (layer 23) - `Water Lines` (layer 20) - Verified live Texas starter query for `Political Jurisdictions`: - `where=ST_FIPS='48'` - `outFields=OBJECTID,DFIRM_ID,POL_NAME1,POL_NAME2,POL_NAME3,CO_FIPS,ST_FIPS,COMM_NO,CID` - `returnGeometry=false` - optional `resultRecordCount=` - Verified live Texas starter query for `Levees`: - `where=DFIRM_ID LIKE '48%'` - `outFields=OBJECTID,DFIRM_ID,LEVEE_ID,LEVEE_NM,LEVEE_TYP,WTR_NM,LEVEE_STAT,OWNER,DISTRICT` - `returnGeometry=false` - optional `resultRecordCount=` - Expose the first useful API routes immediately after the adapter lands: - `src/app/api/water/fema/nfhl/route.ts` - `src/app/api/water/fema/nfhl/political-jurisdictions/route.ts` - `src/app/api/water/fema/nfhl/levees/route.ts` - `src/app/api/water/fema/nfhl/counties/route.ts` - Route behavior: - `/api/water/fema/nfhl` returns normalized metadata plus county-relevant layers - `/api/water/fema/nfhl/political-jurisdictions?limit=N` returns the first live Texas jurisdiction slice - `/api/water/fema/nfhl/levees?limit=N` returns the first live Texas levee slice - `/api/water/fema/nfhl/levees?county=` filters levees by Atlas county using NFHL political-jurisdiction `dfirmIds`; this is a practical first county filter before polygon math - `/api/water/fema/nfhl/counties` returns Atlas county joins built from political jurisdictions - Use a plain ArcGIS REST query URL builder with bounded options: - `where` - `outFields` - `returnGeometry` - optional `resultRecordCount` - always `f=pjson` - Test the adapter with compact sample metadata and mocked route tests instead of live network fixtures. - Before hardcoding a Texas query, verify the live layer metadata once; layer 22 currently exposes `POL_NAME1/2/3`, `CO_FIPS`, `ST_FIPS`, `COMM_NO`, and `CID`; layer 23 currently exposes `LEVEE_ID`, `LEVEE_NM`, `LEVEE_TYP`, `WTR_NM`, `LEVEE_STAT`, `OWNER`, and `DISTRICT`. - The first useful county join is not geometry-heavy: map NFHL political-jurisdiction records onto Atlas counties by combining `ST_FIPS + CO_FIPS`, padding `CO_FIPS` to 3 digits, then resolving that full FIPS through `county-lookup.ts`. - Do not start with real polygon overlay math; use joined jurisdiction counts as the first county-level floodplain footprint proxy. ## Regional authority scouting beyond LCRA - When expanding beyond LCRA, prioritize authorities that already expose one of these surfaces: - ArcGIS Open Data / Hub - ArcGIS Experience or WebMap viewers that likely sit on queryable layers - OneRain dashboards - explicit lake-level / stream-level / rainfall / water-quality endpoints or tables - Verified strong next targets for Atlas Texas, in practical ingest order: 1. GBRA — verified ArcGIS Hub (`opengbra-gbra.hub.arcgis.com`), Hydrolakes, water-quality and Clean Rivers surfaces 2. SARA — verified ArcGIS Open Data (`sariverauthority-sara-tx.opendata.arcgis.com`), Floodplain Viewer, Water Quality Viewer, Watershed Master Plan Viewer, River Basin Report Card 3. Edwards Aquifer Authority — not a river authority, but highly relevant; verified Environmental Data Portal (`data.edwardsaquifer.org`) and aquifer-conditions surface 4. TRA — verified Clean Rivers, lake/river data, flood-planning, water-storage, and ArcGIS experience surfaces 5. SJRA — verified OneRain dashboards plus flood-management and lake-condition pages 6. SRA — verified SE Texas rain, lake levels, stream levels, water-quality monitoring, and Clean Rivers pages 7. LNVA — verified canal-water-quality, Clean Rivers, and OneRain surfaces 8. TRWD — not a river authority, but operationally similar; verified OneRain dashboards, lake-level resources, and ArcGIS map app 9. BRA — reservoir/lake maps verified, but machine-readable surface still needs a deeper pass 10. UCRA — relevant by basin, but public machine-readable surface not yet confirmed - First GBRA slice that shipped cleanly: - adapter file: `src/lib/water/gbra-hydrology.ts` - route files: - `src/app/api/water/gbra/hydrology/major-rivers/route.ts` - `src/app/api/water/gbra/hydrology/gvhs-lakes/route.ts` - tests: - `tests/gbra-hydrology.test.ts` - `tests/gbra-hydrology-routes.test.ts` - Verified GBRA machine-readable source that worked best first: - ArcGIS Hub DCAT feed: `https://opengbra-gbra.hub.arcgis.com/api/feed/dcat-us/1.1.json` - service root discovered from the feed: `https://services7.arcgis.com/Q6vsXnxTnYcWB7qg/arcgis/rest/services/GBRA_Hydrology/FeatureServer` - useful first sublayers: - `1` = Major Rivers - `2` = GVHS Lakes - Live counts verified from the FeatureServer: - Major Rivers: 7 features - GVHS Lakes: 7 features - Normalization shape that worked: - `GbraHydrologyMajorRiver` - `sourceId: 'gbra-hydrology-major-rivers'` - `objectId` from `OBJECTID` - `name` from `GNIS_Name` - `miles` from `Miles` - `shapeLength` from `Shape__Length` - `GbraHydrologyLake` - `sourceId: 'gbra-hydrology-gvhs-lakes'` - `objectId` from `OBJECTID` - `name` from `NAME` - `areaAcres` from `AREA` - `shapeArea` from `Shape__Area` - `shapeLength` from `Shape__Length` - Route behavior that worked: - `/api/water/gbra/hydrology/major-rivers` returns `{ sourceId, featureCount, rivers, freshness }` - `/api/water/gbra/hydrology/gvhs-lakes` returns `{ sourceId, featureCount, lakes, freshness }` - both routes accept optional `?name=` and filter case-insensitively on the normalized `name` field - Freshness/cache wiring that must be updated together: - `src/lib/water/water-source-registry.ts` - `src/lib/water/freshness.ts` - `tests/water-source-registry.test.ts` - Practical GBRA discovery lesson: - GBRA's Hub `api/v3/datasets` endpoint can return noisy cross-tenant/public datasets that are not actually GBRA-specific enough to trust as your first selector. - The ArcGIS Hub DCAT feed was a cleaner path for discovering the real GBRA-owned hydrology service and sublayer URLs. - When a Hub looks noisy, pivot from `/api/v3/datasets` to the domain DCAT feed and then hit the discovered FeatureServer directly. - Recommended ingestion categories across these authorities: - rainfall / stage / flow / reservoir levels - water-quality monitoring and Clean Rivers outputs - floodplain / watershed / master-plan GIS layers - basin report cards or summary indicators ## GBRA water-quality lane - Best reusable finding: do not treat GBRA's ArcGIS water-quality service as the current source of truth for live monitoring. - Verified ArcGIS service: - item id: `d27bbbc66f8e448e8505ac14742c83fa` - service: `https://services7.arcgis.com/Q6vsXnxTnYcWB7qg/ArcGIS/rest/services/Water_Quality_Sample_Stations/FeatureServer` - layer `0` = `CRP Water Quality Monitoring Sites (2019)` - layer `1` = `Surface Water Quality Monitoring Stations (TCEQ)` - Important reality check from live metadata: - ArcGIS item modified timestamp is from 2020 - both layer edit dates are from 2020 - layer 0 is explicitly labeled `(2019)` - use this service only as a legacy inventory / geometry supplement, not as the live program truth - Better live source that worked: - county monitoring pages under `https://www.gbra.org/environmental/water-quality/sites//` - current observation files under `https://waterservices.gbra.org/crp/*.csv` - historical files under `https://waterservices.gbra.org/crp/*.xls` - Example live county page that exposes authoritative links: - `https://www.gbra.org/environmental/water-quality/sites/hays/` - Example live CSV: - `https://waterservices.gbra.org/crp/12672.csv` - verified `Last-Modified` was current in 2026 while ArcGIS metadata was still stale - Best implementation pattern: 1. scrape GBRA county monitoring pages for station metadata 2. extract station id, description, lat/lon, parameter-frequency text, monitoring type, historical XLS URL, current CSV URL 3. use the extracted current CSV URL as the observation source 4. normalize observations from CSV rows into structured records 5. keep ArcGIS inventory optional/secondary for cross-checks only - Important pitfall discovered through trial and error: - do not guess CSV URLs from `stationId.csv` alone - some active stations return `200`, others `404` - the county page table is the authoritative source for whether a current CSV link exists - HTML structure that worked for parsing: - restrict parsing to the GBRA section between: - `Guadalupe River Basin Surface Water Quality Monitoring Program GBRA Monitoring Locations` - `Guadalupe River Basin Surface Water Quality Monitoring Program TCEQ Monitoring Locations` - parse `` rows and table cells rather than trying to infer data from prose - Observation CSV shape that worked: - first line contains station id + station name - header row starts with `Collect Date,Collect Time,Parameter,Reported Result,Units,Parameter Code` - normalize fields into: - `collectedAt` - `parameter` - `reportedResult` - `units` - `parameterCode` - `stationName` - Route set that proved useful: - `/api/water/gbra/quality/sites` - `/api/water/gbra/quality/sites/[stationId]` - `/api/water/gbra/quality/sites/[stationId]/observations` - Test lesson: - `fetchGbraWaterQualitySite()` re-fetches county pages through `fetchGbraWaterQualitySites()`, so mocked fetch sequences must include the county-page responses again before the CSV response - County slug lesson: - `getCountyBySlugOrName()` can return slugs like `hays-county`; normalize them before storing water-quality site `countySlug` values so API/tests stay on Atlas short slugs like `hays` - Scouting method that worked here: 1. Skip Google browser search; it is prone to anti-bot blocks in this environment. 2. Use direct authority homepages and known likely org domains first. 3. Pull links from the homepage and filter for `data|map|dashboard|quality|flood|rain|level|flow|arcgis|onerain|monitor`. 4. If a site blocks raw terminal fetches (for example Cloudflare), use `browser_navigate` + `browser_console` to enumerate links in-page. 5. Rank sources by two axes: machine-readable likelihood and county-joinable usefulness. - Practical finding: some authority sites hide the best ingest surfaces behind branded viewers rather than obvious APIs. ArcGIS hub/viewer URLs and OneRain dashboards are often the fastest route to a usable next endpoint. ## Don’t do first - Don’t start with FEMA statewide polygon ingestion. - Don’t scrape TWDB/TCEQ HTML hubs before enumerating machine-usable sublayers/downloads. - Don’t silently drop unknown-county records; keep them out of county rollups explicitly.