--- name: remote-debug description: > Use when Rex needs to SSH into a client server, diagnose a reported issue, and fix it autonomously. Covers the full cycle: load client context from GBrain → connect → diagnose → fix → write incident report. version: 1.0.0 author: the target organization metadata: hermes: tags: [support, ssh, debug, incident, client] --- # Remote Debug Rex's core workflow for fixing client server issues remotely. ## Trigger - A task appears in the Task Board with Worker Type = AI Agent, assigned to Rex - OR the operator/the founder dispatches Rex directly with a client name + issue description - OR Rex is asked to deploy or set up a service on the the target organization VM (deployment is in scope) ## Step 1 — Load Client Context from GBrain Before touching anything, read the client's history: ``` mcp_gbrain_query("client_name service architecture") mcp_gbrain_query("client_name incident history") ``` Look for: - What software is installed, what version, what stack - Past incidents and how they were resolved - Any known fragile points or warnings ## Step 2 — Load Credentials ### Primary: local secrets file Client SSH/service credentials are stored at: ``` ~/.hermes/secrets/rex/.json # or .env ``` Format: ```json { "host": "x.x.x.x", "port": 22, "user": "deploy", "key_path": "~/.hermes/secrets/rex/keys/.pem" } ``` ### Fallback: external credential store (e.g. Google Sheets URL) If the operator/the founder says "the token/credentials are here" and provides a URL (Google Sheets, Notion, etc.): - **Attempt access first** — use `delegate_task` with `toolsets=["browser"]` to fetch the URL - Do NOT immediately ask for alternatives or say you can't access it — try first, ask only if the delegate returns empty-handed - If access fails, ask for the specific fields needed (don't ask for a full dump) Never log or expose credentials in reports or GBrain entries. ## Step 3 — Connect & Diagnose ```bash ssh -i -p @ ``` Standard diagnostic sequence: ```bash # 1. Check running services systemctl status # 2. Check recent logs (last 200 lines) journalctl -u -n 200 --no-pager # 3. Check disk / memory df -h && free -h # 4. Check process list ps aux | grep # 5. Check recent errors in app logs tail -n 200 /var/log//.log | grep -i error # 6. Check service auto-restart config (critical for crash-recovery questions) systemctl show --property=Restart,RestartSec,ExecStart,RestartMode ``` ## Step 4 — Diagnose → Hypothesize → Test → Fix - State your hypothesis before running a fix - Run the minimal change needed to test the hypothesis - Verify the fix worked before moving on - If a fix requires taking down a service, confirm a rollback path first ## Step 5 — Verify Resolution - Confirm the service is running and healthy - Check logs show no new errors - If possible, trigger a test request to confirm end-to-end functionality ## Step 6 — Write Incident Report After every fix, add a timeline entry to the client's GBrain page: ```python mcp_gbrain_add_timeline_entry( slug="clients/", date="YYYY-MM-DD", summary="", detail="Root cause: ...\nFix: ...\nVerification: ...\nPrevention: ..." ) ``` Then update the Task Board: - Set task status to Done - Write incident summary to Agent Notes field ## Deployment Tasks (Docker Swarm on the target organization VM) When deploying self-hosted services on the VM: 1. **Check port availability first**: `ss -tlnp | grep ':'` 2. **Get internal IP**: `hostname -I | awk '{print $1}'` — GCP metadata endpoint is often org-blocked 3. **Docker Swarm is required** for compose files using `deploy:` keys — `docker compose up` ignores restart policies 4. **`docker stack deploy` does NOT support `--env-file`** — source the env first: ```bash set -a && source .env && set +a && sudo -E docker stack deploy -c docker-compose.yml ``` 5. **After deploy, always check logs** — `docker stack services` showing 1/1 doesn't mean the service is healthy, only that the container started 6. **For stateful services** (Postgres, RabbitMQ): changing passwords after first init requires either dropping volumes (fresh install) or exec-ing in to ALTER USER — the volume bakes in the init password 📎 See `references/plane-docker-swarm.md` for full Plane deployment notes, pitfalls, and the DATABASE_URL placeholder trap. 📎 See `references/plane-self-hosted-ops.md` for RabbitMQ auth fix, Redis cache flush, God Mode bypass shell commands, and the target organization Plane credentials. 📎 See `references/plane-ops.md` for Plane API usage, RabbitMQ fix, Redis cache flush, Django shell bootstrap, and the target organization workspace/project UUIDs. 📎 See `templates/reverse-tunnel.service` for the systemd unit template for a persistent auto-reconnecting reverse SSH tunnel (fill in SITE_USER, SITE_PORT, REMOTE_LISTEN_PORT). 📎 See `references/plane-api-bootstrap.md` for bootstrapping a fresh Plane instance via Django shell (create first user, workspace, API token) and the full REST API usage pattern for creating projects and issues. ## Guiding Non-Technical Operators (Remote Terminal Help) When the client operator is unfamiliar with Linux terminal, give commands **one at a time** and ask for a screenshot after each. Don't batch commands — it's harder to diagnose which step failed. **Common nano sshd_config editing steps** (used frequently for SSH setup): 1. Open file: `sudo nano /etc/ssh/sshd_config` 2. Search: `Ctrl+W` → type the setting name → `Enter` 3. Uncomment a line: `Home` key to start of line → `Delete` to remove `#` 4. Save: `Ctrl+O` → `Enter` (when prompted "File Name to Write" — just press Enter, filename is already correct) 5. Exit: `Ctrl+X` 6. Apply: `sudo systemctl restart ssh` **Key pitfall**: `#PasswordAuthentication yes` is already `yes` — just remove the `#`. Don't change the value. ## Tailscale Access — Cross-Tailnet the target organization VM is on `{{OPERATOR_TAILNET_DOMAIN}}` tailnet (`{{OPERATOR_TAILNET_DNS}}`), IP `{{OPERATOR_TAILNET_IP}}`. Client machines on a **different tailnet are unreachable by default** even if the founder is an admin on their tailnet. > ⚠️ There is also a legacy `{{OPERATOR_TAILNET_ACCOUNT}}` personal tailnet — this is NOT the active one. Always confirm with `tailscale status --json | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['Self']['DNSName'], d['Self']['TailscaleIPs'])"` before assuming which tailnet the VM is on. ### Cross-Account Shared Devices ({{CLIENT_TAILNET_ACCOUNT_EMAIL}} pattern) Example shared client machines can be owned by a separate client tailnet account and shared into the operator tailnet when direct enrollment is not possible. - Shared devices show "Shared in" badge in the primary operator's admin console - They appear at **https://login.tailscale.com/admin/machines** but may NOT appear in `tailscale status` on the VM until routing propagates - `tailscale status` not showing the IP ≠ share not accepted — check admin console first - Even when "Connected" in admin console, SSH port 22 may be firewalled on the client machine - **Stale node trap**: if `{{OPERATOR_TAILNET_NODE}}` shows a different IP in admin console vs `tailscale ip -4` on the VM, there's an old node registered. Delete the stale node from admin console to fix routing. ### SSH Blocked After Tailscale Share Tailscale "Connected" does NOT mean SSH is open. Client Linux machines often have: - SSH service not enabled (`sudo systemctl enable --now ssh`) - UFW blocking port 22 (`sudo ufw allow ssh && sudo ufw reload`) - Tailscale Shields Up blocking inbound peer traffic even while `sshd` is listening Always port-probe first: `nc -zv -w 3 22` — if timeout, verify SSH is listening on the client, then try: ```bash sudo tailscale set --shields-up=false sudo systemctl restart tailscaled sudo systemctl restart ssh ``` If port 22 still times out after Shields Up is disabled and SSH is listening, check the owning tailnet ACL before falling back to reverse tunnel. Always port-probe first: `nc -zv -w 3 22` — if timeout, verify SSH is listening on the client (`sudo systemctl status ssh`, `sudo ss -tlnp | grep ':22'`). If SSH is listening but our probe still times out, ask the operator to run: ```bash sudo tailscale set --shields-up=false sudo systemctl restart tailscaled sudo systemctl restart ssh ``` Then retry direct SSH before moving to ACL changes or reverse tunnels. If it still times out, have the client's Tailscale admin add an ACL for our user/tailnet to reach `:22`. Before attempting SSH to a client machine, verify it appears in `tailscale status`. If it doesn't: - Ping will show 100% packet loss - The fix is **device sharing** (client shares their device to `{{TEAM_EMAIL}}`) — not a tailnet re-auth Key rules: - **Device Sharing** = one-way access, client can't see the target organization infra ✅ preferred - **Re-auth** (`sudo tailscale up --force-reauth`) requires browser interaction — Rex cannot complete autonomously, surface the URL to the operator/the founder. URL expires after ~180s — if it times out, use an **Auth Key** instead (see below) - **Auth Key re-auth (no browser needed)**: the founder generates a one-time key at https://login.tailscale.com/admin/settings/keys → "Generate auth key" (Reusable: No, Ephemeral: No, 1-day expiry) → paste `tskey-auth-...` here → Rex runs: `sudo tailscale up --authkey tskey-auth-xxxxx` This is the preferred path when `--force-reauth` times out - **Stay on `{{TEAM_EMAIL}}`** — already admin on client tailnets; switching to `busycow@` has no confirmed benefit and disrupts existing peers - **Stale ghost nodes cause routing failures** — if `tailscale status` shows a machine with an IP that doesn't match `tailscale ip -4`, there's a stale/duplicate node registered. the founder must delete the old node from the admin console before shared devices will route correctly. Confirm by running `tailscale status --json` and comparing `Self.TailscaleIPs` against what the admin console shows - **Device sharing targets the tailnet, not a node** — when a client shares a device to `{{TEAM_EMAIL}}`, it shares to the tailnet. If there are two nodes registered under that account (e.g. a stale ghost + the active VM), the routing may attach to the wrong node. Always clean up ghost nodes first - **Tailscale DERP relay passes ping (ICMP) but TCP connections may still time out** — a successful `tailscale ping` does NOT guarantee SSH will work. Always follow up with `nc -zv -w 5 ` to confirm TCP is actually reachable - **Client machines authenticated to wrong tailnet** — if a client runs `tailscale up` and logs in with a personal account (e.g. `{{CLIENT_TAILNET_ACCOUNT_EMAIL}}`), their machine joins a different tailnet. The fix is either: (a) `sudo tailscale up --force-reauth` and log in with the correct account, OR (b) share the device from their tailnet to `{{TEAM_EMAIL}}` - **Shared devices not appearing in `tailscale status`**: Connected in admin console ≠ visible to daemon. Shared devices from a different tailnet (`{{CLIENT_TAILNET_ACCOUNT_EMAIL}}` etc.) won't appear until the daemon is refreshed. Try `sudo systemctl restart tailscaled` first. If still missing, the share may have been accepted on the primary operator's personal browser session, not the VM's daemon — use Auth Key re-auth to force a full peer list refresh - **Stale node trap**: If the primary operator's admin console shows two `{{OPERATOR_TAILNET_NODE}}` entries with different IPs, the VM's active IP is `100.77.193.120` — confirm with `tailscale ip -4` on the VM before deleting anything. Deleting the wrong node kills our Tailscale access entirely - **Same-account = same tailnet, no sharing needed**: If client logs their Linux machine into Tailscale using `kevinchan3322@github` (the primary operator's account), the machine joins `{{OPERATOR_TAILNET_ACCOUNT}}` tailnet automatically — no device sharing step required. Confirm which account they used before asking them to share - **`tailscale status Online: True` ≠ ports reachable** — Windows Firewall can block all TCP even when Tailscale handshake is up. Always port-probe with `nc -zv -w 3 ` before concluding a machine is accessible - **Windows RDP port may not be 3389** — clients sometimes use custom ports (e.g. 53389, 50000). Confirm with client before probing. Probe both the custom RDP port AND WinRM (5985) for remote management access - **Tailscale IP blocked ≠ machine unreachable** — some machines have a public domain/port-forward (e.g. `tc42.imprezz.net:50000`). Always probe the domain as a fallback when Tailscale IP times out - **This VM is headless — no RDP GUI** — `xfreerdp` is installed but requires `$DISPLAY`. For Windows-only machines, WinRM (5985) is the only viable remote command path. If WinRM is closed, ask client to enable it via RDP session: `Enable-PSRemoting -Force` ### Tailscale — Linux Client Onboarding (Ubuntu) When a client site needs to install Tailscale on Ubuntu for the first time: ```bash # 1. Install curl -fsSL https://tailscale.com/install.sh | sh # 2. Authenticate (prints a URL — open in browser and log in) sudo tailscale up # 3. Confirm IP tailscale ip -4 ``` **Account matters — check before they authenticate:** - If client logs in with **`kevinchan3322@github`** → machine joins the target organization's tailnet automatically, no sharing needed, appears in `tailscale status` immediately - If client logs in with **their own account** → machine is on a different tailnet, requires device sharing to `{{TEAM_EMAIL}}` - **`tailscale ip -4` returning an IP ≠ authenticated to the right account** — the machine may have a stale auth from a previous account. Always confirm which account was used to log in - To switch accounts: `sudo tailscale up --force-reauth` → browser URL → log in with correct account **Device sharing propagation (cross-account):** - After client shares device to `{{TEAM_EMAIL}}`, it does NOT appear in `tailscale status` or respond to `tailscale ping` immediately - the founder must **accept** the share at https://login.tailscale.com/admin/machines?q=shared%3Ain - Orange **"Shared in"** badge in admin console = share received but may not be accepted/routed yet - After accepting, restart daemon on the target organization VM if still not visible: `sudo systemctl restart tailscaled` - Shared devices from a different tailnet may take 1–2 min to propagate even after acceptance — retry `tailscale ping ` with patience before concluding failure 📎 See `templates/new-support-agent-deployment-prompt.md` for the full self-contained prompt to bootstrap a new Hermes technical support agent with this stack (Feishu gateway, GBrain MCP, [Client] site context, memory seeds, skills list). 📎 See `references/tailscale-cross-tailnet.md` for full diagnostics, [Client] machine IPs, public domain fallback, account decision notes, reverse tunnel setup (password + key variants), port 443 SSH fallback, and Tailscale install check one-liner. 📎 See `references/tailscale-direct-before-tunnel.md` for the preferred order when a shared Linux device pings over Tailscale but SSH times out: direct access first, shields-up/ACL checks, then project-specific reverse tunnel only as fallback. 📎 See `references/client-system-disk-full-recovery.md` for NP/TC Linux AI machine disk-full triage: Tailscale `resolv.conf... no space left`, `/var` drilldown, safe cleanup, and service restart sequence. 📎 See `references/client-system-headless-ai-service.md` for [Client/System] `client_count.service` crash-loop recovery when YOLO/PaddleOCR/OpenCV tries GUI display in headless systemd mode, including `--nosave`, `show_vid`/`cv2.waitKey` guards, `nowStr` fix, and verification steps. 📎 See `references/np-site-access-checks.md` for NP direct Tailscale SSH verification, sudo authority checks, and NP-specific reverse-tunnel fallback using remote port 2223. 📎 See `references/client-system-tailscale-access-guideline.md` for client-facing TC/NP Tailscale access guidance: device sharing model, Windows RDP, Linux SSH, access-control policy, troubleshooting, PDF generation pattern, and direct-access alternatives before reverse tunnel. 📎 See `references/dji-psdk-jetson-orin.md` for DJI M300 / Payload SDK setup on Jetson Orin, including safe isolated workspace, PSDK app credential fields, NVIDIA Jetson CMake build caveat, USB bulk transport error interpretation, and YOLO/TensorRT bridge pattern. 📎 See `references/e-port-dev-kit-jetson-usb-bulk.md` for Matrice 350/300 E-Port Development Kit wiring + Jetson FunctionFS USB bulk setup: adapter port 9 USB 2.0 to Jetson device-mode port, USB ID switch Host, `startup_bulk`, `/dev/usb-ffs/bulk*`, and the `Bad file descriptor` pitfall. 📎 See `references/dji-eport-dev-kit-jetson-connection.md` when an E-Port Development Kit is physically present but PSDK still reports USB bulk `Bad file descriptor`: correct wiring is aircraft E-Port → E-Port coaxial cable → adapter E-Port connector, adapter E-Port switch ON, USB ID switch Host, adapter USB 2.0 data port → Jetson USB-C/device-mode port; XT30-to-USB-C is power only, not PSDK data. 📎 See `references/jetson-dji-psdk-yolo-integration.md` for the safe isolated workflow to prepare a Jetson Orin for DJI M300 PSDK → YOLO/TensorRT integration without disturbing client-created AI code. 📎 See `references/tailscale-client-onboarding.md` for step-by-step Ubuntu client onboarding checklist, common failure table, and [Client] TC/client site B specifics. 📎 See `references/tailscale-client-onboarding.md` for step-by-step Ubuntu client onboarding checklist, common failure table, and [Client] TC/client site B specifics. 📎 See `references/hkrfid-site-inventory.md` for [Client] site map: all 5 ThingsBoard deployments, open ports, ⚠️ exposed PostgreSQL security finding, and TC42 local network layout. 📎 See `references/mapking-hkpc-drone-integration.md` for MapKing/HKPC GHX context: Jetson Orin + DJI M300 PSDK integration objective, initial Tailscale access findings, SSH enablement request, and post-access verification checklist. 📎 See `references/jetson-dji-psdk-yolo-integration.md` for Jetson Orin + DJI M300 PSDK + YOLO/TensorRT audits, no-install software review commands, PSDK discovery, and integration-planning questions. 📎 See `references/hkrfid-site-inventory.md` for [Client] site map: all 5 ThingsBoard deployments, open ports, ⚠️ exposed PostgreSQL security finding, and TC42 local network layout. 📎 See `references/mapking-ghx-orin-integration.md` for MapKing/GHX Jetson Orin access, DJI M300 ↔ PSDK ↔ YOLO/TensorRT integration notes, safe deployment rules, and approved source-backup workflow. ### SSH Reverse Tunnel (when client router blocks inbound) When Tailscale DERP relay passes ping but TCP SSH times out, the client's router is blocking inbound TCP. Use a **reverse tunnel** — the client machine connects OUT to our VM, bypassing all inbound blocks. **Naming rule:** create a project/client-specific tunnel user (e.g. `ghxsite`, `npsite`) instead of reusing an old site name like `tcsite`. Reusing a previous customer/site label confuses stakeholders even if it technically points to the same the target organization VM. **Naming rule:** use a project/site-specific tunnel user on the the target organization VM. Do not reuse old names like `tcsite` for unrelated clients; users notice and it creates confusion. For a GHX/MapKing-style project, create `ghxsite` or another clear site-specific name. If direct SSH still cannot be made to work, use a **reverse tunnel** — the client machine connects OUT to our VM, bypassing all inbound blocks. **Customer-facing naming rule:** use a project/customer-specific tunnel user (e.g. `ghxsite`) instead of legacy names from previous sites (e.g. `tcsite`). A stale name confuses the customer and makes it look like the wrong VM/site is involved. ### Setup on the target organization VM (one-time per site/project) Create a project/site-specific tunnel user. Do **not** reuse old site names such as `tcsite` for unrelated clients; it confuses the founder/client comms even though the user lives on our VM. ```bash # Example for GHX / MapKing project sudo useradd -m -s /bin/bash ghxsite || true echo "ghxsite:" | sudo chpasswd # Enable TCP forwarding in sshd_config sudo sed -i 's/#AllowTcpForwarding yes/AllowTcpForwarding yes/' /etc/ssh/sshd_config sudo sed -i 's/#GatewayPorts no/GatewayPorts yes/' /etc/ssh/sshd_config sudo systemctl restart ssh ``` Never reuse a misleading old tunnel username such as `tcsite` for an unrelated client. It confuses users and makes the support workflow look like it is touching the wrong customer. Use names like `ghxsite`, `npsite`, etc. ### Command for client to run on their machine: Use a **client/project-specific tunnel username and port**. Do not reuse old names like `tcsite` for unrelated customers — even if the account is on our the target organization VM, the name confuses users and looks like the wrong client is involved. Recommended pattern: ```bash # On the target organization VM, create one tunnel user per client/project sudo useradd -m -s /bin/bash # e.g. ghxsite, npsite, tcsite echo ":" | sudo chpasswd # Client runs this on their machine ssh -p 22 -R :localhost:22 -o StrictHostKeyChecking=no -N @34.132.28.199 ``` Example for MapKing/GHX: ```bash ssh -p 22 -R 2224:localhost:22 -o StrictHostKeyChecking=no -N ghxsite@34.132.28.199 ``` - Password: the tunnel user's password from the secure credential store - Terminal will appear to **hang/freeze** — this means the tunnel is **active** ✅ - Once running, Rex SSHs into `localhost:` on the the target organization VM to reach the client machine 📎 See `references/mapking-ghx-tunnel.md` for the session-specific GHX/Jetson reverse-tunnel notes. ### Port selection for outbound SSH from client: - Port **22** — try first (GCP allows inbound 22) - Port **443** — fallback if 22 is blocked outbound; BUT verify port 443 isn't already taken on our VM by another service (`ss -tlnp | grep :443`) before adding it to sshd_config - Port **2244** — requires GCP firewall rule to be opened first (VPC → Firewall → Create rule, TCP 2244 ingress) - ⚠️ GCP firewall blocks all non-pre-configured inbound ports — opening a new SSH port requires a GCP Console firewall rule, not just sshd_config changes ### Plane (self-hosted) — API backend crash-loop (RabbitMQ auth mismatch) When `plane_api` service crash-loops with `amqp.exceptions.AccessRefused: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN`: **Root cause:** `RABBITMQ_PASSWORD` in `/opt/plane/.env` diverged from the password baked into the RabbitMQ Docker volume at first init. Changing `.env` alone does NOT update the running RabbitMQ instance. **Fix (order matters):** ```bash # 1. Reset password in the RUNNING MQ container (not the image — the live one) MQ_ID=$(docker ps --filter name=plane_plane-mq --format "{{.ID}}" | head -1) docker exec $MQ_ID rabbitmqctl change_password plane "NEW_PASSWORD" # 2. Update AMQP_URL and RABBITMQ_PASSWORD in /opt/plane/.env to match # 3. Redeploy stack cd /opt/plane && set -a && source .env && set +a && sudo -E docker stack deploy -c docker-compose.yml plane # 4. Force API service restart to pick up new env docker service update --force plane_api ``` **Critical pitfall:** Every `docker stack deploy` creates a NEW MQ container if the service spec changed. The new container restores password from volume (old value). You must run `rabbitmqctl change_password` AFTER every redeployment that touches the MQ service. **Redis cache pitfall:** After fixing auth, the proxy may still return 502 because it cached the old container IP. Force proxy restart: ```bash docker service update --force plane_proxy ``` **Plane onboarding gate:** Fresh Plane instance shows "God Mode" setup page because: 1. `Instance.is_setup_done = False` in DB 2. Redis caches the `/api/instances/` response — stale False even after DB fix Fix via Django shell: ```bash API_ID=$(docker ps --filter name=plane_api --format "{{.ID}}" | head -1) docker exec $API_ID bash -c "cd /code && python manage.py shell -c \" from plane.license.models import Instance, InstanceAdmin from plane.db.models import Profile, Workspace from django.contrib.auth import get_user_model User = get_user_model() # Fix instance setup gate i = Instance.objects.first() i.is_setup_done = True i.is_signup_screen_visited = True i.save() # Fix user onboarding redirect loop user = User.objects.get(email='{{TEAM_EMAIL}}') profile, _ = Profile.objects.get_or_create(user=user) profile.is_onboarded = True ws = Workspace.objects.first() if ws: profile.last_workspace_id = ws.id profile.save() print('Done') \"" # Then flush Redis REDIS_ID=$(docker ps --filter name=plane_plane-redis --format "{{.ID}}" | head -1) docker exec $REDIS_ID redis-cli FLUSHALL ``` **Plane Django app paths:** - Working dir: `/code` - App module prefix: `plane` (e.g. `from plane.db.models import ...`, `from plane.license.models import ...`) - Run shell: `docker exec $API_ID bash -c "cd /code && python manage.py shell -c \"...\""` — no `-it` flag in non-TTY context ### Multi-Site Health Check (parallel) When the founder asks "check all sites" or "is X, Y, Z working", run checks in parallel using `delegate_task` with one subagent per site. Don't do them sequentially. **Per-site health check sequence:** ```bash # 1. Verify Tailscale peer is actually online (not just registered) tailscale status 2>&1 | grep -E "|" # Look for: "offline, last seen Nd ago" or "rx 0" — these = machine is down # 2. Test management ports (WinRM/RDP) via Tailscale nc -zv -w5 5985 2>&1 # WinRM nc -zv -w5 3389 2>&1 # RDP # 3. Test ThingsBoard web via public domain curl -sk -o /dev/null -w '%{http_code}' https://.imprezz.net curl -skI https://.imprezz.net | head -10 # check SSL + server date # 4. Test PostgreSQL nc -zv -w5 .imprezz.net 5432 2>&1 # 5. Test MQTT (if applicable) nc -zv -w5 1883 2>&1 ``` **Key Tailscale health signals:** - `offline, last seen Nd ago` + `rx 0` = machine is **powered off or network-dead**, not just slow - `active; relay "hkg"` with recent timestamp = connected through DERP relay (TCP may still be blocked) - Machine **not appearing in `tailscale status` at all** = dropped off tailnet entirely (Tailscale disconnected or expired) - `rx 0` = we have never received a single byte from that machine — it's been sending nothing **Severity classification:** - ThingsBoard web DOWN + Tailscale offline = 🔴 P1 full outage (e.g. TWHR 2026-06-11) - ThingsBoard web OK + Tailscale offline = 🟡 P2 management access lost, service still live (e.g. SKS 2026-06-11) - Machine not in tailscale status at all = 🔴 P1 — investigate immediately (e.g. NP 2026-06-11) **Tailscale IP drift warning:** Documented IPs go stale. Always run `tailscale status` to get current IPs before probing. [Client] IPs were all off by 1 (discovered 2026-06-11): the actual IPs end in .15/.24/.53/.39/.34, not .16/.25/.54/.40/.35. See `references/hkrfid-site-inventory.md`. ### [Client/System] / TC / NP Client-Facing Tailscale Access Docs When preparing a client-facing guideline for [Client/System] manual remote maintenance: - Use `vendor` instead of `the target organization` unless the user explicitly wants company-branded wording. - Treat the client device as a **Windows or macOS laptop**. - Treat TC and client site B targets as **Linux AI machines** accessed primarily by SSH over Tailscale. - Do not include Windows RDP instructions unless the user explicitly says Windows site machines are in scope. - State that the Tailscale Console **SSH button** is optional: it creates a browser-based Tailscale SSH session and only works if `sudo tailscale set --ssh` is enabled on the Linux target and ACL/policy permits the user. Otherwise use normal SSH from Windows Terminal/PowerShell or macOS Terminal. 📎 See `references/client-system-tailscale-access.md` for the concise document pattern, wording rules, SSH button explanation, and quality checks. ### Checking if AI/counting program is sending data to PowerBI: When SSH access is unavailable, the fastest check is the PowerBI dashboard itself: - Data timestamps **updating** → program running ✅ - Data **frozen at old timestamp** → program stopped ❌ ### NP/TC Linux Operator Triage — Disk Full + Access Verification When an onsite operator is sharing terminal photos/screenshots from a Linux AI machine, keep commands short and one-at-a-time. For NP/TC counting services, disk-full issues can break Tailscale DNS and service stability before the AI app itself is diagnosable. **If Tailscale health says `no space left on device` or `df -h` shows `/` at 100%:** ```bash df -h / df -ih / sudo du -xhd1 / 2>/dev/null | sort -h ``` - If inodes are low usage but `/` is 100%, the problem is real disk blocks, not file count. - If `/var` is huge, do **not** assume journal logs are the cause. Check: ```bash sudo journalctl --disk-usage sudo du -xhd1 /var | sort -hr sudo du -xhd1 /var/lib 2>/dev/null | sort -hr sudo du -xhd1 /var/log 2>/dev/null | sort -hr sudo find /var -xdev -type f -size +1G -printf '%s %p\n' 2>/dev/null | sort -nr | head -30 | numfmt --field=1 --to=iec ``` - Safe first cleanup only after identifying broad cause: ```bash sudo apt clean sudo journalctl --vacuum-size=1G sudo rm -f /var/crash/* df -h / ``` - Then restart access + AI services: ```bash sudo systemctl restart tailscaled sudo systemctl restart client_count.service sudo systemctl status client_count.service --no-pager ``` **Important access trap:** an operator successfully running `ssh {{CLIENT_SYSTEM_USER}}@` from the NP machine itself only proves the local password works on that machine/tailnet. It does **not** prove Rex/the target organization VM can reach NP. Always resolve and verify the current peer from the VM first: ```bash tailscale status | grep -iE 'client-system|precision|100\.125' NP_IP= tailscale ping --timeout=5s "$NP_IP" nc -zv -w5 "$NP_IP" 22 sudo -n true # after SSH login: confirms whether Rex can amend services without more client commands ``` - `no matching peer` = NP is not shared/routed to the the target organization VM, even if it looks connected on its own Tailscale UI. - If direct TCP/22 succeeds, use direct Tailscale SSH; do not ask the client to run a tunnel. - If direct TCP/22 fails and a reverse tunnel is needed, use NP-specific remote port `2223` so it does not collide with TC's `2222` tunnel. See `references/np-site-access-checks.md`. **Reverse tunnel success signal:** the operator command appears to hang/freeze. That is correct. Rex should confirm by checking the chosen local forwarded port (for NP usually `nc -zv localhost 2223`; for TC usually `2222`) from the VM before trying SSH. ### Tailscale Direct Access Before Reverse Tunnel When Tailscale ping works and `nc -zv -w5 22` succeeds, the network path is healthy. If `ssh` then returns `Permission denied (publickey,password)`, do **not** jump to reverse tunnel — the blocker is authentication, not connectivity. Preferred alternatives before reverse tunnel: - Add the the target organization/Rex SSH public key to the target Linux user’s `~/.ssh/authorized_keys`. - Confirm the correct Linux username/password, or temporarily enable password SSH if explicitly approved. - Enable Tailscale SSH with `sudo tailscale set --ssh` if the owning tailnet ACL allows the maintenance user. - If only status is needed, ask the operator for one-command checks: `systemctl is-active client_count.service && systemctl status client_count.service --no-pager`. Reverse tunnel remains fallback when direct Tailscale TCP/22 is blocked, the client ISP/router blocks inbound, or safe auth cannot be arranged. ### Tailscale Shields-Up Blocks Direct SSH When Tailscale ping works and SSH is listening locally (`sshd` shows `Server listening on 0.0.0.0 port 22`) but remote `nc 22` still times out, check Tailscale shields-up before defaulting to reverse tunnel or ACL work. Operator commands: ```bash sudo tailscale set --shields-up=false sudo systemctl restart tailscaled sudo systemctl restart ssh ``` Then Rex should retry: ```bash nc -zv -w5 22 ssh @ ``` Prefer direct Tailscale SSH/access when possible. Reverse tunnel is fallback only. If using reverse tunnel, create a project/client-specific VM user (e.g. `ghxsite`) instead of reusing stale usernames like `tcsite`, which confuses customers and operators. ### Tailscale Node Offline Triage (rx=0 pattern): When an operator screenshot shows Tailscale health error `no space left on device` or `/` is 100% full, treat disk space as the blocker before repeated service restarts. - `df -h /` at 100% + `df -ih /` low inode usage = real data-block exhaustion, not inode exhaustion - `journalctl --disk-usage` around 1G does **not** explain `/var` in the hundreds of GB — drill into `/var` - Common safe first checks: ```bash sudo du -xhd1 /var 2>/dev/null | sort -hr sudo du -xhd1 /var/lib 2>/dev/null | sort -hr sudo du -xhd1 /var/log 2>/dev/null | sort -hr sudo find /var -xdev -type f -size +1G -printf '%s %p\n' | sort -nr | head -30 | numfmt --field=1 --to=iec ``` - Safe low-risk cleanup before deleting app data: ```bash sudo apt clean sudo journalctl --vacuum-size=1G sudo rm -f /var/crash/* df -h / ``` - After space is freed, restart in order: ```bash sudo systemctl restart tailscaled sudo systemctl restart client_count.service sudo systemctl status client_count.service --no-pager ``` - If operator typed `client_count.service--no-pager`, correct the missing space: `client_count.service --no-pager`. 📎 See `references/client-system-disk-full-recovery.md` for the full operator sequence and deletion cautions. ### Tailscale Node Offline Triage (rx=0 pattern): When `tailscale status` shows `offline, last seen Xd ago` AND `rx 0`: - rx=0 means the machine has received ZERO bytes via the relay — it has not communicated at all - This is NOT a relay issue, NOT a NAT traversal issue — the machine is physically off or lost power/network - No remote fix is possible. Contact site operator to physically check the machine. - Contrast: `rx > 0` with offline = machine was communicating recently but lost connection = could be software ```bash # Quick offline triage tailscale status | grep -E "offline|last seen" # Field layout: IP hostname account OS status... last seen Xd ago tx N rx N # rx=0 = never received any traffic = machine is dead # rx>0 = was alive, recently disconnected = software/network issue ``` If SSH is available: ```bash # Check if python/AI process is running ps aux | grep -iE "python|count|detect|camera" | grep -v grep # Check outbound network connections (data being sent) sudo netstat -tnp | grep -iE "python|established" | grep -v "127.0.0.1" # Find service name if unknown sudo systemctl list-units --type=service --state=running | grep -iE "np|tc|count|ai|camera|detect" # Check logs journalctl -n 50 --no-pager | grep -iE "error|count|detect|send|powerbi|fail" ``` ## Plane (Project Management) — Bootstrap & Repair When Plane is freshly deployed or its API is broken, use these sequences. Full detail in `references/plane-ops.md`. ### API backend crash-loop (ACCESS_REFUSED) Root cause: `AMQP_URL` / `RABBITMQ_PASSWORD` in `.env` diverged from password baked into the RabbitMQ volume after a redeployment. ```bash MQ_ID=$(docker ps --filter name=plane_plane-mq --format "{{.ID}}" | head -1) docker exec $MQ_ID rabbitmqctl change_password plane "NEWPASSWORD" # Then update AMQP_URL and RABBITMQ_PASSWORD in /opt/plane/.env to match # Then redeploy: cd /opt/plane && set -a && source .env && set +a && sudo -E docker stack deploy -c docker-compose.yml plane docker service update --force plane_api ``` ⚠️ After every stack redeploy, the MQ container may re-init with the volume-baked password, not the .env password. Always re-run `rabbitmqctl change_password` after a deploy. ### Plane shows God Mode setup page instead of login Two possible causes: 1. **`is_setup_done=False` in DB** → fix via Django shell (see plane-ops.md) 2. **Redis cached the old response** → this is the likely cause even if DB shows True: ```bash REDIS_ID=$(docker ps --filter name=plane_plane-redis --format "{{.ID}}" | head -1) docker exec $REDIS_ID redis-cli KEYS "*instance*" # confirm cache exists docker exec $REDIS_ID redis-cli FLUSHALL # flush it ``` Always flush Redis AFTER fixing the DB — the frontend reads `/api/instances/` which is Redis-cached. ### Bootstrap first user + workspace + API token (empty Plane install) ```python # Run via: docker exec bash -c "cd /code && python manage.py shell -c \"...\"" from plane.db.models import Workspace, WorkspaceMember, APIToken from plane.license.models import Instance, InstanceAdmin from django.contrib.auth import get_user_model User = get_user_model() # 1. Create superuser u = User(username='kevin', email='{{TEAM_EMAIL}}', display_name='the founder Chan', is_superuser=True, is_staff=True, is_active=True, is_email_verified=True) u.set_password('PASSWORD') u.save() # 2. Create workspace ws = Workspace.objects.create(name='the target organization', slug='dataxquad', owner=u, created_by=u, updated_by=u) WorkspaceMember.objects.create(workspace=ws, member=u, role=20, created_by=u, updated_by=u) # 3. Create API token token = APIToken.objects.create(user=u, workspace=ws, label='rex-agent', created_by=u, updated_by=u) print('TOKEN:', token.token) # save this # 4. Mark instance setup done i = Instance.objects.first() i.is_setup_done = True i.is_signup_screen_visited = True i.save() InstanceAdmin.objects.get_or_create(user=u, instance=i, defaults={'role': 20, 'created_by': u, 'updated_by': u}) ``` Then flush Redis (see above) so the frontend picks up `is_setup_done=True`. ### Plane API URL patterns - Workspaces: `GET /api/v1/workspaces/{slug}/projects/` - Create project: `POST /api/v1/workspaces/{slug}/projects/` - Create issue: `POST /api/v1/workspaces/{slug}/projects/{project_id}/issues/` - Instance check: `GET /api/instances/` (note: NOT `/api/v1/instances/`) - Auth header: `x-api-key: plane_api_...` - Django app label for models: `plane.db.models`, `plane.license.models` 📎 See `references/plane-ops.md` for full Plane deployment notes and the the target organization workspace details. 📎 See `references/plane-project-api-workflow.md` for creating Plane projects, enabling modules, creating issues, linking module issues, and verifying counts through the Plane REST API. 📎 See `references/plane-ops.md` for full Plane deployment notes and the the target organization workspace details. 📎 See `references/plane-project-ops.md` for creating Plane projects, modules, issues, Pages, and Views. Important: API-token `/api/v1` works for projects/issues/modules, but Pages/Views may require Django shell through the `plane_api` container because `/api/...` app routes can reject API tokens. 📎 See `references/plane-ops.md` for full Plane deployment notes and the the target organization workspace details. 📎 See `references/dji-psdk-jetson-orin.md` for the isolated Jetson Orin + DJI PSDK + YOLO/TensorRT integration workflow, including the NVIDIA Jetson PSDK build-dir trap, credential requirements, board-photo identification limits, and plugged-in USB/UART transport triage. 📎 See `references/client-source-backup-github.md` for approved client source-code backup to the target organization VM + private GitHub, including large-file exclusions and token hygiene. ## Pitfalls - ❌ Never run `rm -rf`, `DROP TABLE`, or stop critical services without a rollback plan - ❌ Never assume the issue is the same as last time — always read logs fresh - ❌ Never skip the incident report — it's how the team learns and how Rex builds memory - ❌ Don't write `.env` values using shell variable substitution (`${VAR}`) in bash — produces artifacts like `RAB...}`. Write literal values or use Python/`sed`. - ❌ Don't silently fix security findings (e.g. exposed PostgreSQL ports) — these are client disclosures, escalate to the operator/the founder before touching anything - ✅ If you can't identify root cause with confidence, flag for human review rather than guessing - ✅ If the fix requires a client-facing message or decision, stop and escalate to the operator/the founder - ✅ For Docker deployments: `docker stack services` 1/1 means container started, not healthy — always follow up with `docker service logs` to confirm the app is actually running - ❌ **Plane RabbitMQ password drift**: after any stack redeploy, MQ container may revert to volume-baked password even if `.env` was updated. Always run `rabbitmqctl change_password` on the live container AFTER deploying. Never assume `.env` change alone fixes MQ auth. - ❌ **Plane Redis cache masks DB fixes**: if you fix `is_setup_done` in the DB but the UI still shows the setup page, Redis is serving a cached response. Run `docker exec redis-cli FLUSHALL` after every DB-level Plane fix. Cache key is `:1:/api/instances/`. - ❌ **Plane 502 after API container restart**: the Caddy proxy holds a stale container IP. Fix: `docker service update --force plane_proxy` — forces fresh DNS resolution of `api:8000`. - ✅ **Plane API instance endpoint is `/api/instances/` not `/api/v1/instances/`** — the v1 prefix does NOT apply to the license/instance endpoints. Wrong prefix = 404. - ✅ **Plane Django shell**: use `docker exec bash -c "cd /code && python manage.py shell -c \"...\""` — no `-it` flag (no TTY in terminal tool), working dir must be `/code`, app prefix is `plane.db.models` and `plane.license.models`. - ❌ Don't use `sudo passwd ` interactively in a terminal() call — it fails with "Authentication token manipulation error". Use `echo ":" | sudo chpasswd` instead - ❌ **Plane API path is `/api/instances/` not `/api/v1/instances/`** — the license/instance endpoints are mounted at `/api/instances/`, not under `/api/v1/`. Using the wrong prefix returns 404. - ❌ **Plane module issues endpoint is `/module-issues/` not `/issues/`** — use `POST .../modules/{id}/module-issues/` with `{"issues": [...]}` to link issues to a module. - ❌ **Plane modules not enabled by default** — must PATCH the project first: `{"module_view": true}` before creating modules, otherwise returns `{"non_field_errors": ["Modules are not enabled for this project"]}`. - ✅ **Plane User model requires `username` field** — `User.objects.create_superuser()` needs `username` kwarg. Use `User(username=..., email=..., ...)` + `u.set_password(...)` + `u.save()` pattern instead. - ✅ **client site A SSH is port 22, not 2222** — port 2222 is only used when a reverse tunnel is active. Native SSH is always port 22. Confirm with `nc -zv -w3 22` before assuming port. - ❌ Never paste SSH private keys in Lark group chat — they get redacted by the system. Use password auth as fallback, or send keys via WhatsApp/Signal out-of-band - ✅ When client ISP blocks port 22 AND 2222 for reverse tunnel — add port 443 to our VM's sshd_config and tell client to use `-p 443` in the ssh command. Port 443 is almost never blocked - ✅ `tailscale ping` returning pong via DERP does NOT mean TCP/SSH will work — DERP relays ICMP but may drop TCP. Always probe with `nc -zv -w 5 ` before concluding SSH is reachable - ✅ When probing Windows machines, always scan beyond standard ports — clients may use custom RDP ports (e.g. 53389, 50000). When Tailscale IP is fully blocked, try the public domain (`.imprezz.net`) as fallback before concluding the machine is unreachable - ✅ ThingsBoard ([Client] client stack): web UI is at `https://.imprezz.net`, REST API needs auth token, health endpoint `/api/v1/health` returns 404 on these deployments - ✅ When all inbound TCP is blocked (Tailscale DERP + router + ISP all blocking), use **reverse tunnel**: client SSHes out to our VM, Rex connects via `localhost:2222`. See `references/tailscale-cross-tailnet.md` → Reverse Tunnel section for full setup - ❌ **Never send SSH private keys in Lark group chat** — use WhatsApp, Signal, or a secure file share. Group chats may have multiple members - ✅ `tailscale ping` returning pong via DERP does NOT mean SSH will work — DERP relays ICMP but may not forward TCP reliably. Always test with `nc -zv -w 5 22` before concluding SSH is open - ✅ When admin console shows same hostname with two different IPs (stale node trap): run `tailscale ip -4` on the VM to confirm the active IP, then identify which console entry matches — the other is safe to delete. **Never delete without confirming which is active** - ❌ **`tailscale ping` success does NOT mean SSH works** — ping uses ICMP, SSH uses TCP. DERP relay passes ICMP but may not forward TCP. Always probe with `nc -zv -w 5 22` to confirm SSH reachability separately - ❌ **UFW inactive ≠ port is open** — if UFW is off but SSH still times out, the block is at the ISP/router level or Tailscale DERP relay. Try changing SSH port to 443 to bypass ISP blocks (2222 is not reliably open) - ✅ **client site A ({{CLIENT_SYSTEM_A_TAILSCALE_IP}}) SSH key + password auth both rejected** — confirmed 2026-06-09. Neither `id_ed25519` nor `pc_key` nor common passwords work directly. The only reliable access path is the **reverse tunnel**: TC machine runs `ssh -R 2222:localhost:22 tcsite@34.132.28.199`, then Rex connects via `localhost:2222` on the the target organization VM. Always set up the persistent tunnel service (systemd + SSH key) when you do get access so you don't lose it. - ✅ **client site A native SSH port is 22** — confirmed 2026-06-09. Port 2222 on our the target organization VM is the *reverse tunnel entry point* (when the TC machine tunnels in), not TC's own SSH port. Don't confuse these: `nc -zv {{CLIENT_SYSTEM_A_TAILSCALE_IP}} 22` should succeed when Tailscale is up. - ✅ **client site A ({{CLIENT_SYSTEM_A_TAILSCALE_IP}}) SSH port is 22, not 2222** — confirmed via `nc -zv`. Memory and old notes referenced 2222 but that was the reverse-tunnel forwarded port, not the native SSH port. Always port-probe before assuming: `for port in 22 2222; do nc -zv -w3 $port 2>&1; done` - ✅ **Tailscale "offline, last seen Xd ago" + rx=0 = machine is physically off** — this combination means the machine has not sent OR received any traffic via Tailscale. Not a relay issue, not a NAT issue — the machine is gone. No remote fix possible. Contact site operator to physically check. `rx=0` alone is the key signal: if the relay had connected at all, rx would be non-zero. - ✅ **NP machine drops off `tailscale status` entirely** — when NP ({{CLIENT_SYSTEM_TAILSCALE_IP}}) disappears from `tailscale status` output while TC ({{CLIENT_SYSTEM_A_TAILSCALE_IP}}) is still visible, NP is offline or Tailscale service died. SSH will time out. Ask operator to check power + run `sudo systemctl restart tailscaled`. Cannot diagnose remotely. - ✅ **Tailscale IPs for [Client] sites are all one less than early documentation** — confirmed 2026-06-11. All 5 CMP-PC IPs end in N-1 vs old docs. Always verify with `tailscale status | grep ` rather than trusting static records. - ✅ **ThingsBoard can still serve via public domain even when CMP-PC Tailscale is offline** — the public domain routes via ISP, not Tailscale. A site can show HTTP 200 on HTTPS while the CMP-PC Tailscale node shows `offline, last seen Xd`. Always check both independently. - ✅ **Parallel site health checks via delegate_task** — spawn all 5 sites simultaneously, each as a leaf subagent with `toolsets=["terminal"]`. Each checks: (1) `tailscale status | grep `, (2) `curl -sk -o /dev/null -w '%{http_code}' https://.imprezz.net`, (3) `nc -zv -w5 5985`, (4) `nc -zv -w5 5432`. Full results return in ~50s instead of ~4 min sequential. - ✅ **BWS, TC42, TKO66 WinRM (5985) is actually open via Tailscale** — inventory previously said "all inbound ports firewalled" but 2026-06-11 checks confirmed WinRM responds on all three. RDP custom ports remain blocked. Update any notes that say these sites have no management access. - ❌ **`{{CLIENT_SYSTEM_USER}}` password on client site A is not the known stored candidate passwords tried earlier** — both rejected. Public key auth also fails. Reverse tunnel is the only currently-working path into client site A. When SSH auth fails completely, immediately pivot to asking operator to run the reverse tunnel command rather than trying more passwords - ✅ **NP machine ({{CLIENT_SYSTEM_TAILSCALE_IP}}) can disappear from `tailscale status` entirely** — confirmed 2026-06-11. When a shared device drops off the tailnet, `nc` to port 22 will time out AND the IP won't appear in `tailscale status`. This is different from "offline, last seen Nd ago" (still registered but unreachable). Fix: contact operator to restart Tailscale on the machine (`sudo systemctl restart tailscaled`) or re-authenticate - ✅ **"offline, last seen Nd ago" + rx=0 in tailscale status = machine is powered off**, not just disconnected. "rx 0" means we have never received a single byte — the machine has been contributing nothing to the tailnet. Physical check at site is required - ✅ **ThingsBoard can keep serving after the CMP-PC Tailscale session drops** — the TB process runs independently. Tailscale offline ≠ ThingsBoard offline. Always test the public HTTPS endpoint separately from the Tailscale management probe - ✅ **`sshpass` for non-interactive password SSH attempts** — use `sshpass -p "" ssh @` to test password auth without interactive prompt. Faster than `ssh` when cycling through credential candidates - ❌ **`tailscale status` on VM won't show shared devices until the share is accepted on the CORRECT tailnet** — if VM is on `{{OPERATOR_TAILNET_DOMAIN}}` tailnet, the share must be accepted there. Shares accepted on old/personal tailnets (e.g. `{{OPERATOR_TAILNET_ACCOUNT}}`) won't appear - ✅ **Stale node check before any tailnet debugging** — if admin console shows same hostname twice with different IPs, run `tailscale ip -4` on VM to confirm which node is live. Never delete without confirming — deleting the live node kills Tailscale access entirely - ✅ **New Linux client SSH onboarding**: see `references/tailscale-cross-tailnet.md` → "New Linux Client Machine — SSH Setup Checklist" for the full sequence including nano editing tips for non-technical operators - ✅ **SSSS site access requirements** — to give Rex direct access to the 5 [Client] Windows machines (BWS, SKS, TC42, TKO66, TWHR): (1) each machine shares its Tailscale device to `{{TEAM_EMAIL}}`, (2) the founder accepts at https://login.tailscale.com/admin/machines?q=shared%3Ain, (3) operator runs `Enable-PSRemoting -Force` in PowerShell Admin on each machine. SKS and TWHR already had WinRM open as of 2026-06-10; BWS, TC42, TKO66 were firewall-blocked.