---
name: crawler-health-check
description: >
  Check Promptwatch AI crawler activity, top crawled pages, and sitemap crawl progress.
  Use when diagnosing crawler access, robots.txt blocking, sitemap coverage, AI crawler
  drops, or whether GPTBot, ClaudeBot, PerplexityBot, and similar agents can read a site.
---

# Crawler Health Check

## Prerequisites

Requires the Promptwatch MCP server. Add to your agent config:

```json
{
  "mcpServers": {
    "promptwatch": {
      "url": "https://server.promptwatch.com/mcp",
      "headers": { "Authorization": "Bearer YOUR_API_KEY" }
    }
  }
}
```

Get your API key at https://app.promptwatch.com -> Settings -> API Keys. Run `setup-promptwatch-mcp` if needed.

## Tools

- `list_projects`: resolve project.
- `get_crawler_trend`: crawler activity over time.
- `get_top_crawler_pages`: most crawled pages.
- `get_sitemap_progress`: sitemap crawl status.

## Workflow

1. Resolve project.
2. Pull crawler trend for the last 30 days and previous 30 days.
3. Pull top crawled pages.
4. Pull sitemap progress.
5. Identify which crawlers are present or absent.
6. Check whether crawled pages match the site's important GEO pages.

## Flag Issues

- Crawl volume drops more than 30% week over week.
- Key content pages absent from top crawled pages.
- GPTBot, ClaudeBot, PerplexityBot, or other expected crawlers absent.
- Sitemap progress stalled or missing important URLs.
- Crawlers mostly hit low-value pages instead of category, comparison, docs, or content pages.

## Output

Produce:

1. Crawler activity summary and trend.
2. Crawler breakdown by agent type.
3. Top crawled pages and whether they are strategically useful.
4. Sitemap progress and gaps.
5. Technical recommendations: robots.txt, sitemap, internal linking, page speed, or blocked routes.

If no crawler data exists, recommend verifying Promptwatch tracking, sitemap submission, robots.txt access, and server logs.