--- name: internal-link-graph description: Build the internal link graph for a site, run PageRank-style authority distribution, detect orphan pages, and recommend new internal links via embedding-based semantic similarity (not keyword matching). Crawls the site (sitemap-driven, respects robots), extracts internal links per page, constructs a directed NetworkX graph, computes weighted PageRank, identifies the top sources of authority sink (high-PageRank pages with too few outbound internal links) and the orphans (high-quality content with no inbound internal links). Recommends contextual links by embedding paragraphs from candidate target pages and finding semantically related passages in source pages where a natural link insertion fits. Beats Visibly AI's keyword-matching internal linking by being intent-aware. TRIGGER on "internal linking", "internal link graph", "PageRank", "orphan pages", "internal link recommendations", "site architecture", "link equity", "siloing", "topical clusters". version: "1.0.0" category: analysis platforms: - CLAUDE_CODE --- # Internal Link Graph & Semantic Linking You build the internal link graph and recommend new internal links based on semantic content fit, not keyword density. Internal linking is the highest-leverage SEO control surface that doesn't require new content — and it's typically under-managed because manual audits don't scale past 500 pages. ============================================================ === PRE-FLIGHT === ============================================================ - [ ] **Site URL + sitemap**: `https://example.com/sitemap.xml` or root URL for sitemap discovery. - [ ] **Crawl budget**: number of pages (typical 500-50,000). Above 50k, plan for distributed crawl (Scrapy-cluster or Cloudflare Browser Rendering API). - [ ] **Output target**: actionable internal linking opportunities + site architecture report. - [ ] **Embedding model**: `text-embedding-3-small` (OpenAI, $0.02/1M tokens) OR self-hosted `all-mpnet-base-v2` (sentence-transformers). - [ ] **JavaScript rendering**: required for SPAs / heavy CSR sites. Use Playwright. CPU-bound; budget accordingly. Recovery: - No sitemap: discover via robots.txt + `/sitemap.xml` + `/sitemap_index.xml` + linked-from-root crawl. - Crawl blocked: respect — surface and stop. Don't try to bypass. ============================================================ === PHASE 1: CRAWL === ============================================================ Generate `crawler.py` using `httpx` + `selectolax` for HTML parsing (faster than BS4): ```python class Crawler: def __init__(self, site_root: str, max_pages: int, render_js: bool = False): self.site_root = normalize(site_root) self.max_pages = max_pages self.render_js = render_js # Playwright if True self.seen: set[str] = set() async def crawl(self) -> dict[str, PageData]: seeds = await self._sitemap_urls() results = {} async for url in self._bfs(seeds): html = await self._fetch(url) results[url] = self._extract(url, html) if len(results) >= self.max_pages: break return results def _extract(self, url, html) -> PageData: return PageData( url=url, title=..., h1=..., word_count=..., internal_links=[normalize(href) for href in ... if same_domain(href)], content_text=..., # for embedding content_paragraphs=[...], # for link-position recommendation ) ``` Respect robots.txt. Default crawl-delay 1s. User-agent: `Mozilla/5.0 ... (link-graph-builder; contact: support@yourdomain)`. VALIDATION: Crawl completes within budget. Internal link extraction includes ≥ 95% of `` elements visible in source. ============================================================ === PHASE 2: GRAPH CONSTRUCTION === ============================================================ Generate `graph.py`: ```python import networkx as nx def build(pages: dict[str, PageData]) -> nx.DiGraph: G = nx.DiGraph() for url, p in pages.items(): G.add_node(url, title=p.title, word_count=p.word_count, indexed=p.indexed, noindex=p.noindex) for url, p in pages.items(): for target in p.internal_links: if target in pages: if G.has_edge(url, target): G[url][target]["count"] += 1 else: G.add_edge(url, target, count=1) return G # PageRank with edge weights (multi-link from same source counts more) pr = nx.pagerank(G, alpha=0.85, weight="count") ``` VALIDATION: Graph node count == crawled-page count. Edges count > nodes (typical: 10-50 edges per node). ============================================================ === PHASE 3: AUDIT METRICS === ============================================================ For each page: - **PageRank** (internal authority). - **In-degree** (unique pages linking IN). - **Out-degree** (unique internal pages linked TO). - **Avg path depth from homepage** (BFS shortest path). - **Click depth from homepage** (BFS — important: Google crawls deepest pages least often). Generate flags: | Flag | Definition | Action | |---|---|---| | **Orphan** | In-degree = 0 (no inbound internal links) | Add ≥ 2 contextual links from related pages | | **Near-orphan** | In-degree = 1 from a low-PR page | Add 2-3 inbound links from cluster | | **Authority sink** | High PR + low out-degree | Add outbound links to deeper cluster pages | | **Dead-end** | Out-degree = 0 (rare for non-leaf pages) | Add related-content section | | **Too deep** | Click depth > 4 from homepage | Restructure nav or add internal-link shortcut | | **Over-linked** | Out-degree > 200 | Possible footer/template noise diluting equity | Output `link_graph_audit.csv` with all flags. VALIDATION: Audit identifies orphans + authority sinks against fixture site. ============================================================ === PHASE 4: SEMANTIC LINK RECOMMENDATION === ============================================================ The real differentiator: recommend NEW internal links based on semantic similarity, not keyword matching. For each candidate target page (priority = orphans, high-intent pages, conversion pages): 1. Embed the target page's title + H1 + summary. 2. Embed each paragraph of every source page (one embedding per paragraph). 3. Find paragraph-level cosine matches ≥ 0.72 (tunable). 4. Rank candidate source paragraphs by similarity × source PageRank. 5. For the top 10 paragraphs per target, propose an anchor-text + insertion point. Generate `link_recommendations.md`: ```markdown ## Target: /pricing Current in-degree: 2 ### Recommended new inbound links: 1. **From `/blog/pricing-models`** (PR: 0.0042, similarity: 0.81) - Suggested anchor: "our transparent pricing" - Insertion point: 3rd paragraph, after "...different pricing models exist." - Context: "Most SaaS uses per-seat pricing. [our transparent pricing] uses usage-based..." 2. **From `/blog/saas-buyer-guide`** ... ``` Limit recommendations to 5-10 per target — anything more pollutes naturalness. VALIDATION: For a sample target page, recommended links are demonstrably relevant (not "/about" linking to "/pricing" via "click here"). ============================================================ === PHASE 5: SILO / CLUSTER ANALYSIS === ============================================================ Detect topical clusters via: - Community detection (Louvain or Leiden algorithm on the link graph). - Or semantic clustering (k-means on page embeddings). - Reconcile: a healthy cluster has high internal cohesion AND high embedding similarity. Output `topical_clusters.md`: - Cluster N: {topic label generated from top 5 highest-PR pages' titles} - Pillar candidate: page with highest within-cluster PageRank - Supporting pages: rest of cluster - Recommended pillar-to-supporting links: any missing - Recommended cross-cluster bridges: 1-2 per cluster (not more — diffuses focus) VALIDATION: Cluster labels are recognizable to a human looking at the URLs. ============================================================ === PHASE 6: IMPLEMENTATION HELPER === ============================================================ For each recommended link, generate either: - A **markdown patch** if the source page is in a markdown CMS (Hugo, Jekyll, Astro, Next.js MDX). - A **CMS-specific snippet** if WordPress / Webflow / Squarespace. - A **JSON manifest** for the editorial team to insert via their CMS UI. Output `link_patches/` directory with one file per source page containing all queued link insertions. VALIDATION: Patches are syntactically valid (markdown renders, JSON parses). ============================================================ === PHASE 7: REPORTS === ============================================================ ``` internal-link-graph/ ├── README.md ├── data/ │ ├── crawl.db │ └── graph.gexf # importable to Gephi for visual analysis ├── reports/ │ ├── audit_summary.md │ ├── link_graph_audit.csv │ ├── orphan_pages.csv │ ├── authority_sinks.csv │ ├── link_recommendations.md │ ├── topical_clusters.md │ └── pagerank_distribution.png └── patches/ └── {hash}.patch.md ``` VALIDATION: All reports render. GEXF opens in Gephi correctly. ============================================================ === SELF-REVIEW === ============================================================ - **Complete**: Crawl + graph + audit + semantic linking + cluster + patches? - **Robust**: Respects robots, handles JS-rendered sites, scales to 50k pages? - **Clean**: Recommendations have specific anchor text + insertion points, not just URL pairs? - **SEO-credible**: Would a technical SEO who's read SEO Theory recognize this as legit? Common gap: keyword-anchor-text repetition (linking 50 pages to /pricing with anchor "pricing" looks manipulative). Vary anchor text from the surrounding context. ============================================================ === LEARNINGS CAPTURE === ============================================================ `~/.claude/skills/internal-link-graph/LEARNINGS.md`. ============================================================ === STRICT RULES === ============================================================ - Never propose 50+ new internal links pointing to the same target with identical anchor text. Anchor diversification is a real ranking signal. - Never recommend links from unrelated pages just to lift in-degree. Topical fit > raw link count. - Never crawl past robots.txt. Respect. - Always preserve the crawl history. The graph is a longitudinal artifact — month-over-month changes inform architecture decisions. - Always include both PageRank-weighted AND semantic recommendations. PR-only = old-school; semantic-only = misses authority flow.