--- name: ops-desktop description: Autonomous desktop + browser control via the desktop-act MCP companion. Acquires an isolated noVNC desktop session, takes screenshots, clicks, types, scrolls, and runs the optional autonomous act() loop. First run auto-bootstraps the desktop-act server into a per-user cache. argument-hint: "[goal text | 'status' | 'list' | 'release [session_id]']" allowed-tools: - Bash - Read - AskUserQuestion - ToolSearch - mcp__desktop-act__acquire_desktop - mcp__desktop-act__release_desktop - mcp__desktop-act__list_desktops - mcp__desktop-act__list_windows - mcp__desktop-act__launch_app - mcp__desktop-act__screenshot - mcp__desktop-act__observe - mcp__desktop-act__click - mcp__desktop-act__keypress - mcp__desktop-act__type_text - mcp__desktop-act__scroll - mcp__desktop-act__act - mcp__desktop-act__act_step - mcp__desktop-act__batch - mcp__desktop-act__status effort: medium maxTurns: 40 --- # OPS ► DESKTOP `/ops:desktop` drives a real GUI desktop (Linux noVNC pool, partial macOS / Windows support) through the `desktop-act` FastMCP server. Use it for tasks that browser-only automation can't reach: native dialogs, OS file pickers, multi-window flows, screen-recording verification. The MCP tool schemas are **deferred**. Load them with `ToolSearch` on demand: ``` ToolSearch select:mcp__desktop-act__acquire_desktop,mcp__desktop-act__screenshot,mcp__desktop-act__click,mcp__desktop-act__type_text,mcp__desktop-act__release_desktop ``` ## Runtime Context Before any route runs, resolve: 1. **Plugin root**: `${CLAUDE_PLUGIN_ROOT}` — used to locate the launcher. 2. **MCP launcher**: `${CLAUDE_PLUGIN_ROOT}/mcp-servers/desktop-act-launcher.py` — auto-discovers an installed desktop-act marketplace or, if absent, clones from `$DESKTOP_ACT_REPO` (default `https://github.com/Lifecycle-Innovations-Limited/desktop-act.git`) and bootstraps a venv on first run. 3. **noVNC default**: `http://:6081` — primary desktop. `acquire_desktop()` provisions extra sessions on `:6082+`. 4. **Mobile / SSH mode**: if `$SSH_CONNECTION`, `$SSH_CLIENT`, or `$SSH_TTY` is set, emit compact text-only output per Rule 7 (no banners, no tables, plain lines). The noVNC URL is the one piece the user definitely needs — print it once on a copy-able line. ## Routing table Parse `$ARGUMENTS` and route. First token decides: | First arg | Route | | ------------------ | ------------------------------------------------------------------------------------- | | (empty) / `status` | Print MCP status, default noVNC URL, active sessions, and platform support note | | `list` | `mcp__desktop-act__list_desktops` — show pool state | | `release ` | `mcp__desktop-act__release_desktop(session_id)` — free a session | | `release-all` | Release every session this skill has open this turn | | anything else | Treat as a **goal** — acquire → observe → drive → verify → release | Anything else: treat the whole argument string as the goal. --- ## Route — `status` (default) 1. Probe the MCP via `mcp__desktop-act__status` (auto-load schema first). 2. Print compact summary — plugin connected? venv ready? pool size? default URL? 3. If the MCP is **not reachable**, the launcher likely needs to bootstrap. Show: - one-line install command (`/plugin marketplace add && /plugin install desktop-act`) - or the env override (`export DESKTOP_ACT_COMMAND=/path/to/run.sh`) - or the bootstrap repo (`export DESKTOP_ACT_REPO=https://...`) 4. Mobile/SSH mode: drop banners, output 4–6 plain lines. ## Route — `list` ``` ToolSearch select:mcp__desktop-act__list_desktops ``` Call it. Format as `session_id · display · vnc · last_used`. Linux only renders full info; macOS/Windows fall back to a "limited platform" note. ## Route — `release ` / `release-all` Per Rule 5 (destructive actions): `release_desktop` tears down the X server. Ask before bulk release: ``` AskUserQuestion: Release session abcd1234? [Release] [Skip] ``` For `release-all`, batch confirm once. --- ## Route — **goal** (default for free-form $ARGUMENTS) The autonomous loop. Use this when Sam types `/ops:desktop open the AWS console and screenshot the ECS cluster page`. ### Step 1 — Acquire ```python # Load schemas ToolSearch select:mcp__desktop-act__acquire_desktop,mcp__desktop-act__screenshot,... session = mcp__desktop-act__acquire_desktop() session_id = session["session_id"] display = session["display"] # e.g. ":51" novnc_url = session["novnc_url"] # e.g. http://host:6082 ``` Print the noVNC URL on a line by itself so the user can watch live. On mobile, the URL is the most useful thing — surface it FIRST. ### Step 2 — Observe ```python shot = mcp__desktop-act__screenshot(session_id=session_id) Read(file_path=shot["path"]) # let Claude see the screen ``` If you need window-level metadata before clicking blindly: ```python windows = mcp__desktop-act__list_windows(session_id=session_id) ``` ### Step 3 — Drive Two modes: **A. Streaming primitives** — full reasoning visibility, per-action transparency. Preferred for sensitive flows (anything touching credentials, billing pages, destructive UIs). ```python mcp__desktop-act__launch_app(session_id, app="firefox") mcp__desktop-act__click(session_id, x=640, y=400) mcp__desktop-act__type_text(session_id, text="ECS clusters") mcp__desktop-act__keypress(session_id, key="Return") mcp__desktop-act__scroll(session_id, direction="down", amount=3) mcp__desktop-act__screenshot(session_id) # verify ``` **B. Autonomous loop** — hands the goal to `act()` which runs claude-agent-sdk against the OAuth-bundled CLI (no API key needed; rides Claude Max). Faster but less observable. ```python result = mcp__desktop-act__act( session_id=session_id, goal="$ARGUMENTS", max_steps=20, ) ``` For multi-step macros, prefer `batch` (single round-trip) or `act_step` (one micro-step at a time with explicit verification). ### Step 4 — Verify Always `screenshot` and `Read` the path once at the end before claiming the goal is done. If the verification screenshot doesn't show the expected state, iterate or escalate to the user. ### Step 5 — Release **Always** release in the same turn, even on failure: ```python mcp__desktop-act__release_desktop(session_id=session_id) ``` If the user wants to keep the session for follow-up actions, ask first via `AskUserQuestion`: ``` Keep session abcd1234 alive for follow-up? [Keep] [Release] ``` --- ## Cross-platform notes | OS | Status | Notes | | ------- | -------- | --------------------------------------------------------------------- | | Linux | Full | X11 + Xvnc + websockify + python-xlib. Multi-desktop pool live. | | macOS | Partial | Server runs; native X11 calls no-op. Browser tools still work. | | Windows | Partial | Server runs via Python launcher. Native automation requires manual. | On non-Linux platforms, prefer Kapture (`mcp__kapture__*`) or Playwright for browser-only tasks. The launcher prints a clear "limited platform" notice if it can't bring up a full session. ## Safety - **Outbound comms** (Rule 6): if a goal involves sending email/messages/forms, stage the draft and ask before pressing Send. Per-message approval, never batch. - **Destructive UI clicks**: anything that says "Delete", "Terminate", "Force quit" routes through `AskUserQuestion` first per Rule 5. - **Credentials**: never type passwords into a goal string — leave it for Sam to fill in the live noVNC viewer. - **Session isolation**: each invocation of `/ops:desktop` gets its own `session_id`. Don't pass session IDs between concurrent agents or skills. ## Examples ``` /ops:desktop # status /ops:desktop list # pool state /ops:desktop release abcd1234 # free session /ops:desktop open the AWS ECS console and screenshot the healify-api cluster /ops:desktop launch Firefox and navigate to https://finops.lifecycleinnovations.limited ``` ## $ARGUMENTS $ARGUMENTS