---
name: tailscale-pihole-dns-routing
description: This skill should be used when a host or device on a network with both Tailscale and Pi-hole seems to bypass Pi-hole's ad-blocking despite DNS still resolving fine, when deciding whether to use Tailscale's tailnet-wide "Override DNS servers"/Global Nameservers setting (including whether to enable it for roaming ad-blocking), when a device's DNS silently breaks after that Override toggle is turned off, when Pi-hole silently rejects/resets DNS queries arriving over a Tailscale interface, or when systemd-resolved needs to be configured so a host gets Pi-hole for general DNS while keeping Tailscale MagicDNS working for *.ts.net hostnames. Trigger phrases include "tailscale override dns servers", "pihole bypassed tailscale", "resolv.conf generated by tailscale do not edit", "magicdns not resolving publicly", "systemd-resolved pihole tailscale", "Domains=~. resolved.conf", "tailscale accept-dns per device", "DNS=~. wildcard route", "pihole-dns-client playbook", "host missed by dns overhaul", "apt-get update failed after systemd-resolved install", "SSH_CLIENT tailscale IP over LAN hostname", "magicdns wrong answer right after tailscaled restart", "pihole listeningMode LOCAL", "dns.listeningMode ALL", "pihole rejects tailscale query connection reset", "roaming ad-blocking over tailscale", "nftables flush ruleset wipes tailscaled", "cloudflare backup nameserver tailscale unreliable", "tailscale global nameserver backup order", "ssh could not resolve hostname", "getent hosts not found but dig works", "ResolveUnicastSingleLabel", "single-label hostname resolution fails", "bare hostname ssh resolve failure ansible", "resolvectl query not found LLMNR", "restart of systemd-resolved failed exit status", "dns using direct mode directManager", "tailscale apt upgrade broke dns", "systemctl restart tailscaled did not fix dns", "resolv.conf plain file instead of symlink".
---

# Tailscale + Pi-hole: getting both to coexist correctly

## The core conflict

Three requirements that fight each other on a network running both Tailscale (for remote access,
e.g. `tailscale serve`) and Pi-hole (for ad-blocking):

1. Server hosts (never roam) should always resolve through Pi-hole, with correct per-host
   attribution in Pi-hole's dashboard.
2. Roaming personal devices (laptop, phone) — this is a genuine **decision point**, not a fixed
   requirement: either they should get Pi-hole *only* while on the home LAN (no added latency or
   broken resolution while traveling, no filtering while away), *or* they should get Pi-hole
   everywhere, including while roaming, accepting a small latency cost in exchange. See "Decision:
   should roaming devices get Pi-hole at all?" below — don't assume one answer without asking.
3. `*.ts.net` MagicDNS hostnames must keep resolving everywhere, on every device that needs them.

**Tailscale's DNS model, as of tailscale 1.98.8, doesn't have a clean way to satisfy all three at
once with one tailnet-wide switch:**

- **Global Nameservers + "Override DNS servers"** (admin console → DNS tab) is a **tailnet-wide,
  all-or-nothing toggle**. When on, every device that accepts Tailscale's DNS config routes all
  non-MagicDNS queries through the configured nameserver — including while roaming. Real-world
  latency varies a lot by connection path: other users report ~90ms typical, up to ~1s at p90 when
  a direct WireGuard peer connection can't be established and traffic falls back to a DERP relay
  (see [tailscale/tailscale#18340](https://github.com/tailscale/tailscale/issues/18340)) — but a
  live measurement on one tailnet (2026-07-19, phone tethered as a hotspot to force a genuine
  off-LAN path rather than testing same-LAN) found only ~20-25ms added per uncached lookup once a
  direct peer connection was established (~184ms round-trip to home). **Don't assume either figure
  without measuring on the actual network in question** — see "Measuring the real latency cost"
  below for the method. DNS caching (both OS-level and Pi-hole's own) means this cost is paid once
  per new domain, not per request.
- **Per-device "Accept DNS"** (`tailscale set --accept-dns=false`) is the only per-device lever,
  but it's all-or-nothing *per device*: turning it off exempts that device from Global
  Nameservers, but **also disables that device's ability to resolve MagicDNS `*.ts.net` peer
  hostnames locally**. Confirmed live: `*.ts.net` hostnames do **not** resolve correctly via plain
  public DNS — a query for e.g. `grafana.<tailnet>.ts.net` against a normal recursive resolver
  returns an unrelated public IP (or nothing), not the real tailnet address. So a host that needs
  to reach other tailnet members by hostname can't just drop Tailscale DNS.
- **Per-device Global-Nameserver override doesn't exist yet** — open feature request
  [tailscale/tailscale#18379](https://github.com/tailscale/tailscale/issues/18379).
- **Tailscale does not reliably treat a second Global Nameserver as backup-only.** If you're
  tempted to add a second nameserver (e.g. Cloudflare `1.1.1.1`) alongside Pi-hole for resilience
  when Pi-hole/home is unreachable: an open Tailscale GitHub issue and user reports both describe
  it racing multiple configured nameservers rather than trying them in strict order, and
  iOS/macOS specifically have a known bug where fallback to an alternate nameserver doesn't
  reliably trigger when the first one goes down
  ([tailscale/tailscale#12677](https://github.com/tailscale/tailscale/issues/12677),
  [tailscale/tailscale#5397](https://github.com/tailscale/tailscale/issues/5397) — a feature
  request for backup-only nameservers, confirming it doesn't exist yet). Adding one risks some
  queries silently bypassing Pi-hole's filtering unpredictably, defeating the point. Don't add a
  second Global Nameserver as a "just a fallback" assumption — check whether Tailscale has since
  shipped real backup-only support before doing this.

## Decision: should roaming devices get Pi-hole at all?

Two valid architectures, depending on whether roaming ad-blocking is worth a small, real latency
cost. Ask before assuming either:

| | Override **off** (roaming = unfiltered) | Override **on**, pointed at Pi-hole's Tailscale IP |
|---|---|---|
| Roaming DNS latency | None — uses whatever local DNS the network hands out | +20-25ms per uncached lookup (measured), possibly more on a bad connection path |
| Ad-blocking while roaming | None | Full — same gravity list, same filtering as home |
| Setup complexity | Low — MagicDNS + per-host systemd-resolved for servers only | Low, but **requires** the Pi-hole `listeningMode` fix below or it silently fails |
| Failure mode if Pi-hole/home goes down while roaming | N/A, not depended on | DNS breaks for roaming devices until Tailscale reconnects or you disable Override (no reliable automatic backup — see above) |

If the answer is "Override off" (or "servers only, not personal roaming devices"), skip to
**"Servers: always through Pi-hole, correctly attributed"** below — the rest of this section
doesn't apply. If the answer is "Override on, want roaming ad-blocking," continue here.

### Pointing Global Nameservers at Pi-hole's LAN IP silently doesn't work off-network

A very easy mistake: entering Pi-hole's LAN IP (e.g. `192.168.50.53`) as the Global Nameserver.
That's only reachable from devices actually on that LAN. Off the LAN (traveling), with no
Tailscale subnet router advertising that subnet (check via `tailscale status` — devices offering
**routes**, not just **exit nodes**, are a different thing), the nameserver is simply unreachable
and DNS quietly breaks for roaming devices with Override on. Point it at Pi-hole's own
**Tailscale IP** instead (`tailscale status | grep pihole`) — that resolves from anywhere in the
tailnet, since Pi-hole itself runs `tailscaled`.

### Pi-hole silently rejects the query anyway: `dns.listeningMode`

Even with the right IP, a second, independent bug can make this look like it's still not working:
Pi-hole's `dns.listeningMode` defaults to (or is commonly set to) `LOCAL`, which only answers
queries whose source IP falls inside the **same subnet** as one of Pi-hole's own interfaces.
Pi-hole's `tailscale0` interface is a single-host `/32` (e.g. `100.124.205.55/32`) — there's no
subnet to match against, so **every query arriving over Tailscale gets rejected**, correctly per
LOCAL mode's own definition, not a bug in Tailscale or a stale-interface-detection issue (a
`pihole-FTL` restart does not fix it — confirmed live, since `tailscaled` and `pihole-FTL` start
within a second of each other on the same boot, ruling out startup-ordering as the cause).

**Symptom to recognize:** a TCP connection to Pi-hole's Tailscale IP on port 53 completes the
handshake fine (`nc -zv <ip> 53` succeeds), but the actual DNS query gets `connection reset`
(`dig @<ip> +tcp` → `communications error ... connection reset`) or simply times out over UDP.
This looks like a firewall block but isn't — it's the DNS server application itself refusing the
query after accepting the connection.

**Fix:**
```bash
# on the Pi-hole host
sed -i 's/listeningMode = "LOCAL"/listeningMode = "ALL"/' /etc/pihole/pihole.toml
systemctl restart pihole-FTL
```
or idempotently via `pihole-FTL --config dns.listeningMode ALL` (read current value first with
`pihole-FTL --config dns.listeningMode` — it prints just `ALL`/`LOCAL`, easy to compare in an
Ansible `when:` clause).

`ALL` means "permit all origins, accept on all interfaces, respond to all" — Pi-hole's own docs
warn to make sure it isn't exposed to the internet when using this mode. If there's no port
forward to Pi-hole (the normal case for a homelab), this is already true — but **scope it
explicitly with a firewall rule anyway** rather than relying on "nothing forwards to it today":

```
# /etc/nftables.conf — inet filter table, input chain
iifname "lo" accept
udp dport 53 ip saddr { 192.168.50.0/24, 100.64.0.0/10 } accept   # LAN + tailnet CGNAT range
tcp dport 53 ip saddr { 192.168.50.0/24, 100.64.0.0/10 } accept
udp dport 53 ip6 saddr fd7a:115c:a1e0::/48 accept                  # tailnet ULA range (check yours)
tcp dport 53 ip6 saddr fd7a:115c:a1e0::/48 accept
udp dport 53 drop
tcp dport 53 drop
```
(Get your tailnet's actual IPv6 ULA prefix from `ip addr show tailscale0` on any peer — it's
consistent tailnet-wide.) Validate syntax before applying: `nft -c -f <file>`.

### Gotcha: reloading nftables wipes `tailscaled`'s own firewall rules

If the host is Tailscale-connected, **`systemctl restart nftables` (or any `nft -f` of a file
starting with `flush ruleset`) also deletes `tailscaled`'s own separately-managed iptables-nft
tables** (`ip filter`/`ip6 filter`/`ip nat`/`ip6 nat` — anti-spoofing rules, and if the host offers
exit-node/subnet-router service, its NAT masquerade chain). `flush ruleset` is a global operation
across every table in every family, not scoped to the table your own config declares. These are
**not** reprogrammed automatically, and basic connectivity (`tailscale status`, ping) keeps working
regardless since WireGuard itself doesn't depend on them — which makes the gap easy to miss.

**Fix: restart `tailscaled` immediately after any nftables reload on a Tailscale-connected host**
to force it to reassert its own tables. In Ansible, order the handler that restarts `tailscaled`
strictly after the handler that reloads nftables. Verify with `nft list ruleset | grep -c 'table
ip '` before/after (tailscaled's tables are all commented `# Warning: ... managed by iptables-nft,
do not touch!`) — don't assume a clean `nft -c -f` syntax check plus a successful reload means
nothing else was affected.

### Measuring the real latency cost

Don't trust a single at-home test — your Mac/phone will negotiate a **direct WireGuard peer
connection** to any tailnet host reachable on the same LAN, which is essentially free latency-wise
and doesn't represent the real roaming case at all. To get an honest number:

1. **At home first (control):** `dig @<pihole-tailscale-ip> <domain> +stats` for a handful of
   domains, cold and warm — establishes the same-LAN best case.
2. **Force a genuine off-LAN path:** tether the test device to a phone hotspot instead of home
   Wi-Fi (`tailscale ping <pihole-tailscale-ip>` will show whether it's using `DERP(<region>)` —
   slow, hundreds of ms — or found a direct path with a real RTT). Rerun the same `dig` benchmark.
3. Compare against a plain public resolver (`dig @1.1.1.1 <domain>`) from the same hotspot
   connection as a baseline for "what does DNS cost with no home dependency at all."

This is the only way to know whether the tradeoff is actually worth it on a given network — the
number varies a lot by how far the round trip to home actually is and whether a direct peer path
is achievable at all.

### Verifying the fix actually works: prove the path, don't trust the config

**A query resolving correctly doesn't prove it went through Tailscale** — two ways this bites:

- **A pre-existing Tailscale subnet route can make a query to Pi-hole's *LAN* IP silently succeed
  even while genuinely off-LAN**, if some other host in the tailnet advertises that LAN subnet as
  a route and the querying device has "Use Tailscale Subnets" enabled (the default). This routes
  through an extra hop (subnet router → LAN) rather than the direct peer connection to Pi-hole's
  own Tailscale IP, giving noisier/worse latency numbers that can look like a broken test rather
  than a routing quirk. Check `netstat -rn` / `ip route` for an unexpected route to the LAN subnet
  via a tailscale interface before trusting a "LAN IP still works while roaming" result.
- **The only reliable proof a query went through the tailnet path is checking the source IP Pi-hole
  actually saw it from**, not just "did it resolve." Fire a uniquely-named, guaranteed-uncached
  query from the device, then check on the Pi-hole host:
  ```bash
  grep '<the-unique-name>' /var/log/pihole/pihole.log
  # look for: query[...] <name> from <source-ip>
  ```
  A source IP matching the device's **Tailscale** IP (`tailscale status`, not its LAN IP) confirms
  the query genuinely traveled over the tunnel. Use the live log, not the long-term SQLite query
  log — the latter lags real traffic by a flush interval and can show zero rows for a query that
  already landed, making a real success look like a failure.

Run the full check on both the primary roaming devices (verify a normal domain resolves, a known
blocklisted domain returns `0.0.0.0`, and MagicDNS `*.ts.net` names still resolve) — do this with
the device's Wi-Fi off (forced onto cellular) for phones, not just at a desk on the home network.

## Servers: always through Pi-hole, correctly attributed

This applies regardless of what you decide above for roaming personal devices — server-class
hosts (LXCs, VMs, anything that never roams) should always resolve through Pi-hole with correct
per-host attribution.

The load-bearing behavior that makes this work independent of the Override decision: **with
Override off, Tailscale on Linux registers only the MagicDNS split into whatever DNS manager it
detects, and leaves all other queries to that manager.** If no DNS manager is present, Tailscale
takes over `/etc/resolv.conf` directly ("direct mode") — and then there's no fallback at all for
general queries (see the next section). Installing systemd-resolved gives Tailscale a manager to
register into, so general DNS goes wherever systemd-resolved is configured to send it.

### Why turning off "Override DNS servers" can silently break DNS entirely (not just visibility)

If a host is in Tailscale "direct mode" — `/etc/resolv.conf` reads `nameserver 100.100.100.100`
only, with a header saying `# resolv.conf(5) file generated by tailscale ... DO NOT EDIT THIS
FILE BY HAND -- CHANGES WILL BE OVERWRITTEN` — turning off Override removes its only forward
target. The host doesn't fail to resolve (Tailscale still answers *something*, resolving general
queries some other way), but it stops reaching Pi-hole **entirely**, not just anonymously.
Confirmed via a uniquely-named, guaranteed-uncached query fired from the host, checked for on
Pi-hole's side — zero hits, not even attributed to a fallback IP. This is a strictly worse
failure mode than the visibility-only gap roaming devices get, because it's silent — the host
still has working internet, so nothing *looks* broken.

**Don't try to fix this by hand-editing `/etc/resolv.conf` or adding a second `nameserver` line.**
Two independent reasons it won't work:
1. That file is regenerated by Tailscale; a manual edit gets overwritten.
2. Even if it weren't managed, glibc's resolver only falls through to a second `nameserver` line
   when the *first* one times out — not based on whether the first one gave a "wrong" answer.
   Tailscale's `100.100.100.100` always answers (just not via Pi-hole), so a second line would
   never actually get used.

### The fix: install/configure systemd-resolved, pin it to Pi-hole, keep MagicDNS

**Reusable Ansible playbook**: `playbooks/pihole-dns-client.yml` in `homelab-ansible`
(deployed nowhere separately — run directly with `ansible-playbook playbooks/pihole-dns-client.yml
--limit <host>`), idempotent, handles the ordering gotcha below automatically. Use this instead
of the raw commands unless working on a host outside that inventory. Added 2026-07-19 after the
original grafana/homepage/jellyfin/pbs fix was applied by hand over SSH and never captured as
code — which is part of why `n8n` and `ansible-ctrl` (built/joined after that fix) were silently
missed by it, and why `immich` had the same gap despite being built afterward too. Re-run it
against the whole fleet periodically, or add every new host to it as a standard build step.

For a host currently in Tailscale direct mode, the manual equivalent:

```bash
mkdir -p /etc/systemd/resolved.conf.d
cat > /etc/systemd/resolved.conf.d/10-pihole.conf <<'EOF'
[Resolve]
DNS=192.168.50.53
FallbackDNS=
Domains=~.
ResolveUnicastSingleLabel=yes
EOF

# Write the drop-in BEFORE installing/starting the package -- confirmed live 2026-07-19 that
# Debian's systemd-resolved postinst immediately takes over /etc/resolv.conf and starts the
# daemon as part of installation. Doing this step after leaves a real window (long enough to
# break `apt-get update` on the very next task) where the daemon is running with no upstream
# DNS configured at all.
apt-get install -y systemd-resolved
systemctl enable --now systemd-resolved
systemctl restart systemd-resolved   # only needed if the package/service already existed

# Hand /etc/resolv.conf to systemd-resolved so Tailscale can detect it as a DNS manager
ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
tailscale set --accept-dns=true
systemctl restart tailscaled   # forces Tailscale to re-detect the DNS manager and register
                                 # the ts.net split into it, instead of overwriting resolv.conf
```

`Domains=~.` is the wildcard route — it makes the *global* scope (Pi-hole) win for every domain,
so a competing per-link resolver (e.g. one handed out over `eth0` via DHCP) can't steal general
queries. Tailscale's own `ts.net` registration is more specific and still wins for MagicDNS.
Leave `FallbackDNS` empty — a configured fallback silently defeats the whole point (a query that
times out against Pi-hole would otherwise fall through to a public resolver, bypassing blocking).

If a host already runs systemd-resolved (check `systemctl is-active systemd-resolved` /
`resolvectl status` first) but its upstream isn't Pi-hole, skip the install step — just write the
same drop-in and restart. Don't be alarmed if `resolvectl status` still shows a per-link entry
(e.g. `eth0` → the router) alongside the `Global` scope's `192.168.50.53` — the `Domains=~.`
wildcard on the Global scope wins for actual resolution regardless; the per-link entry is inert
noise from DHCP, not a competing active path. Confirm with a real query, not just the status
output.

**Connection safety:** `systemctl restart tailscaled` is the one step that can sever your own
session — and **dialing a target by its plain LAN IP is not sufficient proof the connection
itself is going over the LAN.** Confirmed live 2026-07-19: SSH sessions in this fleet addressed
by LAN IP (both from an external Mac to `ansible-ctrl`, and from `ansible-ctrl` on to `n8n`,
same physical subnet) both showed a Tailscale `100.x` address as `$SSH_CLIENT`'s source, meaning
the traffic was actually flowing through Tailscale's routing layer under the hood despite the
LAN-looking destination. Don't infer safety from which address you typed — check the actual
`$SSH_CLIENT` (or equivalent) on the *target* host before assuming a synchronous restart is
safe. **Default to the delayed/detached form always**
(`systemd-run --on-active=5s --unit=<name> systemctl restart tailscaled`), not just when
"only reachable over Tailscale" — it's harmless even on a genuinely local connection and removes
the need to correctly judge connection type case by case. Reconnect and re-verify after it
completes rather than assuming success.

**Post-restart timing:** a query fired immediately after the delayed `tailscaled` restart
completes can transiently return a wrong/public answer for a `*.ts.net` MagicDNS name (observed:
resolved via the `eth0` link instead of `tailscale0`, giving back an unrelated public IP) even
though the fix is genuinely correct — Tailscale takes a few extra seconds beyond the restart
itself to fully re-register the DNS split after reconnecting to the control plane. Don't
conclude MagicDNS is broken from one query taken right after the restart; wait ~10-15s (or just
retry once) before treating a failed magicdns query as a real regression.

### Gotcha: bare single-label hostnames (`pve`, `pihole`) can fail even though the drop-in is correct

systemd-resolved's own default is `ResolveUnicastSingleLabel=no` — it refuses to resolve a bare,
dot-free hostname (`pve`) via real unicast DNS at all, falling back to LLMNR/mDNS only. This is
silently masked on most hosts in this fleet because they run an LLMNR responder themselves and
answer for their own name — `resolvectl query homepage` "works," but the answer came from
`homepage`'s own LLMNR reply (visible as `-- link: eth0` in the output, a link-scoped answer, not
a real DNS one), not from Pi-hole. Hosts that don't run an LLMNR responder — `pve` (the Proxmox
host itself) and `pihole` (the DNS server, not a generic Debian LXC) — get a hard failure instead:
`ssh root@pve` (or any tool going through NSS's `resolve` module — `getent`, most non-`dig`
programs) reports "Name or service not known" even though Pi-hole holds the record and answers it
correctly when queried directly (`dig @192.168.50.53 pve` succeeds). `dig` with no explicit
server also appears to work in this state, but only because it queries `/etc/resolv.conf`'s
`nameserver` directly (Tailscale's own `100.100.100.100` stub, which has no such single-label
restriction) — that's a different, NSS-bypassing code path from what `ssh`/`getent` actually use,
so it doesn't prove the real bug is fixed.

**Confirmed live 2026-07-29** on `ansible-ctrl`: a cross-host automation's `ssh root@pve` call
failed to resolve `pve` at all, reported as "nightly backup summary can't reach pve" even though
every backup job on `pve` was fine — a pure DNS-resolution bug wearing a backup-failure costume.
Fixed by adding `ResolveUnicastSingleLabel=yes` to the drop-in and re-running the playbook, then
**rolled out fleet-wide the same day** (`pbs`, `caddy`, `grafana`, `homepage`, `n8n`, `immich`) —
`pihole-dns-client.yml` now carries the fix by default for any host it's run against.

**A second, independent drift turned up during the fleet-wide rollout**: `pbs`, `caddy`,
`grafana`, `homepage`, and `n8n` all had `/etc/resolv.conf` as a real Tailscale-managed file
(`# generated by tailscale`), not the symlink to `/run/systemd/resolve/stub-resolv.conf` this
playbook is supposed to establish — the same drift `ansible-ctrl` had. This means the
single-label bug had been live on all of them the whole time (masked by their own LLMNR
self-answers), not a hypothetical fleet-wide risk. `ansible-playbook --check --diff` surfaces
this cleanly before the real run: watch for the resolv.conf task showing `state: file → state:
link` rather than `ok`/ no diff. `pihole-dns-client.yml` fixes both the drop-in and the symlink
in the same idempotent run, so a single re-run closes both gaps together.

**Verification note:** `resolvectl query pve` showing `-- link: eth0` in its one-line summary is
*not* by itself a sign of LLMNR — check the full multi-line output (`resolvectl query pve`, no
truncation) for `Information acquired via protocol DNS` vs. a link-scoped LLMNR answer. Several
hosts in this fleet legitimately show `-- link: eth0`/`-- link: tailscale0` for genuine DNS
answers too; the annotation is about the outbound scope, not the protocol used.

### Gotcha: the resolv.conf-symlink drift recurs — it's not a one-time fix, it's any `tailscaled` restart

**Confirmed live 2026-08-02, twice in the same day.** The fleet-wide 2026-07-29 fix (above) is
not durable. `ansible-ctrl` drifted back into "direct" mode after a plain host reboot the
following day (2026-07-31) went unnoticed until the next weekly sweep; separately, `grafana`,
`homepage`, `n8n`, and `caddy` were all *also* found freshly drifted on 2026-08-02, this time
surfaced during a routine `tailscale` package upgrade (1.98.x → 1.98.10) — the `tailscale`
package's postinst does `deb-systemd-invoke restart tailscaled.service` unconditionally, which is
exactly the same restart-races-systemd-resolved trigger as a reboot. **Any `tailscaled` restart
is a recurrence risk on this fleet, not just reboots** — treat every future `tailscale` apt
upgrade as an opportunity for this to resurface, not a routine no-op.

**Diagnostic fingerprint** — `journalctl -u tailscaled --since <restart time>` showing this
sequence means it just fell into the bad state:
```
restarting systemd-resolved...
restart of systemd-resolved failed: exit status 4      # or: signal: killed
dns: [rc=unknown ret=direct]
dns: using "direct" mode
dns: using *dns.directManager
```
`systemd-resolved` itself restarting cleanly seconds later (check `journalctl -u systemd-resolved`
— on a healthy host it shows clean `Stopped`/`Started` pairs, no OOM/kill in `journalctl -k`)
doesn't mean the drift self-heals — `tailscaled` has already committed to "direct" mode for that
process lifetime and won't retry.

**Plain `systemctl restart tailscaled` alone does not fix an already-drifted host** — confirmed by
trying it first on `ansible-ctrl` 2026-08-02 before finding the real fix. Once `tailscaled` has
overwritten `/etc/resolv.conf` with its own plain-file stamp, that file itself is what confuses
its manager auto-detection on the *next* restart too (no more evidence in the file that
`systemd-resolved` ever owned it) — a self-reinforcing loop. The symlink must be restored **before**
the restart, not after:
```bash
rm /etc/resolv.conf
ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
systemctl restart tailscaled   # or the delayed form, see Connection safety above
```
Then verify with `resolvectl domain`/`resolvectl dns` showing the `tailscale0` link scope
populated (not just `Link N (tailscale0):` with nothing after it) — that's the actual signal the
D-Bus registration succeeded, not just that the symlink exists.

**`pihole-dns-client.yml` already encodes this fix correctly** (force-links resolv.conf every run
regardless of prior state, restarts `tailscaled` after) — re-running it against a host is the
standard remediation, faster than doing the manual steps by hand. `ansible-playbook
playbooks/pihole-dns-client.yml --limit <host> --check --diff` first to confirm it's actually
going to touch the symlink (`state: file → state: link`) before applying for real.

**Gap closed 2026-08-09** after recurring a third time (2026-07-29, 2026-08-02, 2026-08-09):
`dns-symlink-watchdog` (`homelab-ansible`, 15-min systemd timer on `ansible-ctrl`, same shape as
`boot-watchdog`/`pihole-dot-failover`) now checks `/etc/resolv.conf` symlink state fleet-wide
every tick via `ansible dns_client -m stat` and re-runs `pihole-dns-client.yml --limit <drifted>`
for real on any drift, Telegram-alerting only on an actual fix or a failed one (30-min cooldown).
The weekly sweep's `resolv_conf_symlink_health` check (added 2026-08-02) stays as a slower
report-only backstop behind it. New `[dns_client]` Ansible inventory group is the source of truth
for which hosts this applies to (`ansible-ctrl pbs caddy grafana homepage n8n immich` — not
`pve`/`pihole`, neither runs systemd-resolved); `pihole-dns-client.yml` was switched from
`hosts: all` to that group so a future run can't accidentally target them. **Validated 2026-08-09**
by a routine tailscale 1.98.10→1.102.2 fleet upgrade (4 more `tailscaled` restarts across
`pihole`/`pbs`/`grafana`/`ansible-ctrl`) — zero drift found on any host, confirming the watchdog
is holding the fix between restarts rather than drift just not having happened to strike yet.

**Validated again 2026-08-30 at real mass-drift scale**: a full `pve` host reboot restarts every
guest simultaneously, which restarts every guest's `tailscaled` simultaneously — the first time
this watchdog was tested against more than one or two hosts drifting at once. 5 of the 7
`dns_client` hosts drifted in that single reboot; the watchdog's very next 15-minute tick caught
and fixed all 5 in one `pihole-dns-client.yml --limit` run, confirmed via `journalctl -u
dns-symlink-watchdog.service` and a live re-test (`getent hosts grafana.tail922cee.ts.net`
returning the real tailnet IP, Tailscale Serve URLs back to 200). No manual intervention
needed — good evidence the design holds at fleet-reboot scale, not just single-host/single-apt-
upgrade drift.

**`pve` and `pihole` themselves are exempt, not "still broken"**: neither runs systemd-resolved
at all, so `ResolveUnicastSingleLabel` doesn't apply. `pve` resolves via plain glibc NSS pointed
straight at Pi-hole; `pihole` resolves via Tailscale's own MagicDNS search-domain expansion. Both
already resolve single-label names correctly by a different mechanism — don't try to force the
systemd-resolved drop-in onto either.

**`dfw` (and any host outside the `homelab-ansible` fleet) is out of scope for this whole
pattern, not a missed host**: it's a separate Vultr VPS managed by its own standalone
`dfw-ansible` repo, was never onboarded onto Pi-hole/systemd-resolved-split-DNS, and doesn't run
`systemd-resolved` at all (`systemctl is-active systemd-resolved` reports `inactive`,
`resolvectl` isn't even installed). Its plain Tailscale-managed `/etc/resolv.conf` (pointed at
`100.100.100.100`, MagicDNS's own stub) is the **correct, intended** configuration there, not a
drifted fallback — confirmed 2026-08-30 after briefly misreading `dfw`'s `resolv.conf` as the
same drift bug mid-investigation (same "generated by tailscale, do not edit" header text as a
genuinely drifted homelab host, which is what triggered the false match) before checking whether
it even runs systemd-resolved and finding real hostname resolution (`n8n.tail922cee.ts.net`,
external domains) working correctly throughout. Don't apply the `test -L /etc/resolv.conf` check
or this skill's fix to `dfw` or any other non-fleet host without first confirming it's actually
meant to be running systemd-resolved in the first place.

**Acceptance tests to run on every host after this change** (don't just trust that `example.com`
resolves — verify each piece separately, and don't let an LLMNR-answered query fool you into
thinking single-label unicast DNS works):
```bash
resolvectl status | grep -E 'Current DNS Server|DNS Servers|DNS Domain'   # (a) Pi-hole is global
resolvectl query some-other-host.<tailnet>.ts.net                         # (b) MagicDNS intact
resolvectl query example.com                                              # (c) general resolution
dig +short some-known-blocklisted-domain.com                              # (d) ad-blocking: 0.0.0.0
resolvectl query pve                                                      # (e) single-label unicast DNS,
                                                                           #     not LLMNR -- output must say
                                                                           #     "Information acquired via
                                                                           #     protocol DNS", not a
                                                                           #     "-- link: <iface>" suffix
```
Then verify **attribution** on the Pi-hole side by firing a uniquely-named query from the host
and checking it's attributed to *that host's own IP*, not the router's or absent entirely — see
[[pihole-local-dns-records]] for the live-log verification technique (Pi-hole's long-term SQLite
query log lags real traffic by a flush interval; check `/var/log/pihole/pihole.log` instead for
instant confirmation).

### Gotcha: a host that skips this fix entirely can deadlock at cold boot, not just drift

A distinct, worse failure mode from the drift bug above: a host that was **never** brought under
this systemd-resolved setup at all — e.g. provisioned via Terraform/manual build outside the
normal Ansible onboarding flow — has no independent resolver whatsoever. Its only DNS path is
Tailscale's own MagicDNS stub (`100.100.100.100`), and if the tailnet's Global Nameservers point
at tailnet-only IPs (Pi-hole's own Tailscale IP, a second resolver node, etc. — not a public
resolver), that stub can't resolve `controlplane.tailscale.com` until the host is *already*
logged into Tailscale. Chicken-and-egg: it needs DNS to log in, and needs to be logged in for
DNS to work. Confirmed live 2026-09-01 (`agent-vault` LXC, homelab fleet): the host retried
`register request: ... controlplane.tailscale.com` in an endless backoff loop for 20+ minutes
after a host power-cycle, never self-resolving — this is genuinely stuck, not slow.

**Diagnostic fingerprint**: `tailscale status` shows `offline`; `journalctl -u tailscaled` shows
repeated `dns udp query: waiting for response or error from [<tailnet-only IPs>]: context
deadline exceeded` interleaved with failed `register request` attempts, and often a failed
`bootstrapDNS(...)` attempt too (tailscaled's own embedded fallback for exactly this situation,
which can lose a race against the doomed tailnet-DNS queries eating the whole request context
before the bootstrap attempt gets a fair shot).

**Break the immediate deadlock**: temporarily point `/etc/resolv.conf` at a public resolver
(`nameserver 1.1.1.1`) — even though Tailscale's own header says not to hand-edit this file, that
warning is about *drift after the fix is applied*, not about breaking an active deadlock before
any fix exists. `systemctl restart tailscaled` after. It will register with the control plane and
reconnect within seconds once real DNS works.

**Permanent fix**: apply the full systemd-resolved-pinned-to-Pi-hole setup from "The fix" section
above to this host, exactly like any other fleet host — this gives it an independent resolver
that works before Tailscale is connected, closing the deadlock permanently rather than requiring
manual intervention on every future cold boot. Don't stop at breaking the immediate deadlock.

**Variant for a host with no LAN path to Pi-hole at all** (an external VPS reachable only over
Tailscale — e.g. a cloud-hosted agent gateway): Pi-hole's LAN IP isn't reachable from outside the
LAN, and its Tailscale IP requires an active Tailscale connection to reach — using either as the
Global-scope `DNS=` just recreates the same deadlock one layer down. There is no way to keep
Pi-hole filtering/attribution for this host's own general DNS *and* survive a cold boot — pick a
public resolver (`DNS=1.1.1.1`) as the Global scope instead, same `Domains=~.`/
`ResolveUnicastSingleLabel=yes` drop-in otherwise. This is a real, deliberate trade-off (the host
loses Pi-hole ad-blocking/attribution for its own traffic), not a workaround to feel bad about —
same conclusion this skill already reaches for the "roaming devices" decision, and ad-blocking
has no real value for a headless server anyway. Confirmed live 2026-09-01 on two Vultr VPSes
(`dfw`/OpenClaw, `hermes`/Hermes) running an agentic gateway that itself depends on Tailscale
reachability (both route real outbound API calls through a homelab-hosted credential broker) —
both had `CorpDNS=true` and a bare Tailscale-managed `resolv.conf`, identical to the LXC that
triggered this whole finding, and neither had been rebooted since going into that dependency.
Installing `systemd-resolved` here also removes the `resolvconf` package if present (Debian
treats them as alternatives) — check `apt-mark showauto resolvconf` and `apt-get remove --dry-run
resolvconf` first; if it shows `0 to remove` beyond itself, it's dormant (a stale
`dhclient-enter-hooks.d` leftover, not actively managing anything since Tailscale was already
overwriting `/etc/resolv.conf` directly) and safe to let go.

**The real lesson generalizes past DNS**: any host provisioned outside the normal fleet-onboarding
flow (Terraform-only, hand-built, "I'll just get to the Ansible parity later") inherits none of
the fleet's baseline resilience assumptions — not just this DNS fix, but monitoring, backup jobs,
and drift-detection coverage too. It's invisible to every host-list-driven check until someone
explicitly closes that gap. Treat "provisioned outside the normal flow" as a standing flag to
check every touchpoint a normal host gets automatically, not just the one that happened to bite.

## The router-relay trap: DHCP handing out the router's own IP instead of Pi-hole's

Separately from anything Tailscale-related: if a router's DHCP server hands out **its own IP**
as the DNS server (a common consumer-router default — the router runs its own local caching
forwarder, using Pi-hole only as *its own* upstream), every client using that default DHCP
value gets correctly ad-blocked, but Pi-hole's dashboard shows it all attributed to the
**router's IP**, not the originating device — because the router does its own new outbound query
to Pi-hole, sourced from its own address, rather than transparently relaying with the original
client's source IP preserved. This masks per-device visibility for potentially the entire LAN.
Fix: change the router's DHCP **DNS Server** field (distinct from any static IP/reservations
page) to Pi-hole's IP directly, and disable any "also advertise router's IP" option if present —
after a lease renewal, clients query Pi-hole directly and attribute correctly. Static hosts not
on DHCP (e.g. a hypervisor with a fixed IP) need `/etc/resolv.conf` edited directly instead — this
is the one case a plain manual edit is correct and durable, since nothing else manages that file.

## ASUSWRT: "WAN DNS Setting" and "LAN → DHCP Server" DNS fields are two different settings

On ASUS routers (confirmed on an RT-AX1800S, stock firmware `3.0.0.4.386_69195`), **Advanced
Settings → WAN → Internet Connection**'s "WAN DNS Setting" card and **Advanced Settings → LAN →
DHCP Server**'s "DNS and WINS Server Setting" card both show near-identical UI (DNS Server 1/2
fields, an "advertise router's IP" toggle) but control genuinely different things: the WAN page
sets what the *router itself* uses for its own outbound DNS queries; the LAN→DHCP Server page
sets what gets pushed to LAN clients via DHCP option 6. **Setting only the WAN page does nothing
for client DNS redundancy** — clients keep getting whatever the DHCP page has configured (often
just the router's own IP, or a single nameserver), regardless of what WAN shows. Confirmed
2026-08-27 adding a second Pi-hole as DNS2: the WAN page already correctly showed both IPs, but a
client's actual DHCP-assigned nameserver list (`scutil --dns` on macOS) only had one entry until
the *separate* LAN→DHCP Server page's DNS Server 2 field — which had been typed in but never
actually clicked Apply on — was saved. If a client isn't picking up an expected DNS server
change, check both pages independently rather than assuming one config surface covers both, and
verify via a live DHCP lease renewal on a real client, not just the router UI showing saved
values.

## Verifying real failover/resolution behavior on macOS: don't use bare `dig`

`dig` (no `@server` argument) reads `/etc/resolv.conf` directly and does **not** go through
`mDNSResponder` — so it doesn't reflect macOS's actual scoped-resolver configuration (the
multi-nameserver-per-interface setup `scutil --dns` shows, which is what real applications use
via `getaddrinfo`). A `dig`-based failover test can report total failure (`connection timed out;
no servers could be reached`) even when the real system resolver would succeed via a second
configured nameserver. Confirmed 2026-08-27 testing DNS2 failover: bare `dig` timed out
completely after 18s with the primary DNS server stopped, while `dscacheutil -q host -a name
<domain>` — which does go through the real resolver path — succeeded (took ~44s, slow but
correct). Use `dscacheutil -q host -a name <domain>` (or an actual app/browser) for any test
meant to represent real client DNS behavior on macOS, not `dig` by itself. `dig
@<specific-ip> <domain>` (with an explicit server) is still fine for testing one resolver in
isolation — the gap is specifically bare `dig`'s use of `/etc/resolv.conf` as a stand-in for the
system resolver, which it isn't.

## By-name SSH can land on the tailnet address — and a Tailscale SSH host then hangs it forever (2026-09-12)

On a host running systemd-resolved with Tailscale, a bare hostname doesn't resolve consistently.
`getent ahosts pihole` returned the LAN IPv4 and a LAN IPv6, while `resolvectl query pihole` returned
the **tailnet address via `tailscale0`**. Across two replays, 4 of 8 `ssh root@pihole` calls went over
the tailnet.

If the target has **Tailscale SSH on** (`tailscale set --ssh=true` / `RunSSH: true`), a connection
arriving on its tailnet address is taken over by tailscaled. It prints:

```
# Tailscale SSH requires an additional check.
# To authenticate, visit: https://login.tailscale.com/a/...
```

…and **waits indefinitely**:
- `ConnectTimeout` never fires, because TCP already connected.
- `BatchMode=yes` doesn't stop it.
- The only trace is on the target: `journalctl -u tailscaled | grep "handling conn"` shows the
  intercepted connection and its source tailnet IP.

**Consequence:** a scheduled check script that reaches hosts by name, with no per-call or per-job
timeout, hangs on the first such host. A systemd oneshot that never exits also blocks every later run
of that unit, so one host with Tailscale SSH on silences the whole check **indefinitely**. It's
silence, not an alert, and it happens *before* any RunSSH audit further down the script can report it.

**Remedy (all four, not one):**
1. **Bound every scheduled SSH call:** `timeout -k 5 <secs> ssh -o BatchMode=yes -o ConnectTimeout=10
   -o ServerAliveInterval=10 -o ServerAliveCountMax=3 …`, via one shared helper. Set per-call limits
   from a timed normal run.
2. **Turn the hang into a finding:** rc 124 or that banner in stderr → a `fail` naming the host and
   Tailscale SSH interception.
3. **Run the RunSSH audit first**, over LAN IPs only, so its answer lands before anything can stall.
4. **Add a job-level timeout that reports:** run the job under `timeout` *inside* the wrapper that
   records and alerts, not only a unit `TimeoutStartSec`, which kills the wrapper too and leaves no
   record.

Keeping hostnames (rather than switching to LAN IPs) still exercises the route real automation uses,
and interception now shows up as a finding. **To reach a host that has Tailscale SSH on, use its LAN
IP.** When testing, arm the revert from a path that can't itself be intercepted.