--- name: qirabot description: "Drive any GUI with natural language — click, type, extract, and verify on web browsers, Android, iOS, desktop apps, and games — using the Qirabot Python SDK. Use this when the user wants to automate, test, or scrape a user interface by describing elements in plain language instead of CSS/XPath selectors; when driving a mobile app or a native desktop/game where DOM-based tools don't work; or for visual UI verification, screenshots, and RPA. Triggers include: automate a website or app, UI/end-to-end test, fill a form, scrape a page, tap or click a button, verify what's on screen, drive an Android/iOS app, automate a desktop application." license: MIT metadata: author: qirabot --- ## Step 0 — Preflight (always run first) Do NOT write an automation script before the environment checks out. Run: ```bash python scripts/preflight.py browser # or: android | ios | desktop ``` It verifies Python version, `QIRA_API_KEY`, that `qirabot` is importable, and target-specific bits (e.g. `adb devices`). **If it fails, stop and fix what it prints** — don't proceed and debug a half-set-up run. **One interpreter, one source of truth.** Preflight validates *one* Python and, on success, prints its absolute path (`interpreter: ...` and a run line). Run your script with **that exact path**, never a bare `python` — otherwise the run can drift to a different env than the one you just checked (the #1 false-"Ready" trap). Whatever already works (an existing venv, the user's global) is fine — the point is to *reuse the validated one*, not to force a new venv. Bootstrap only when preflight reports something missing. Prefer an isolated venv over the user's global Python (and **re-run preflight with that venv's Python** so it becomes the validated interpreter): ```bash # One backend per project → use the conventional .venv (IDEs auto-detect it): python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate pip install "qirabot[browser]" # → also: playwright install chromium # or qirabot[appium] (iOS: + Appium server & WebDriverAgent) / qirabot[desktop] # or qirabot[airtest] (Android, iOS, window-scoped Windows desktop; Python 3.10-3.12) # CAVEAT: airtest pins numpy<2, which conflicts with the browser/desktop extras. # Only if ONE project genuinely needs airtest AND another backend, give airtest # its own separately-named env instead of sharing .venv: # python3.12 -m venv .venv-airtest && source .venv-airtest/bin/activate # pip install "qirabot[airtest]" # Credentials go in a .env file (the templates call qirabot.load_dotenv() to read # it; a real exported env var still wins). Never hard-code the key in the script. echo 'QIRA_API_KEY=qk_...' > .env # from https://app.qirabot.com # optional self-host: echo 'QIRA_BASE_URL=https://...' >> .env ``` ## Step 1 — Pick a target and start from a template | Target | Template | Extra | |---|---|---| | Web browser (Qirabot launches Chromium) | `templates/browser.py` | `qirabot[browser]` + `playwright install chromium`. Also supports connecting to an existing Chrome via `cdp_url` (e.g. Browserless/Browserbase). | | Android — Airtest (no Appium server, fastest start) | `templates/android.py` | `qirabot[airtest]` (Python 3.10-3.12) | | iOS — Airtest (drives WDA directly, no Appium server) | `templates/ios_airtest.py` | `qirabot[airtest]` (Python 3.10-3.12). Real device needs WDA running on :8100 (typically via `iproxy 8100 8100`). The template launches apps via WDA's `app_launch`, not airtest's `start_app` — the latter routes through go-ios and breaks on iOS 17+. | | iOS — Appium XCUITest (simulators, auto WDA build/sign) | `templates/ios_appium.py` | `qirabot[appium]` + a running Appium server & WDA. Real device (iOS 17+) needs a built/signed WDA; simplest is to leave WDA running on :8100 and reuse it via `webDriverAgentUrl` (the template does this). | | Any Selenium driver, or other Appium targets (you build the driver, then `bind()`) | `templates/bolt_on.py` | `qirabot` + `selenium` / `qirabot[appium]` | | Desktop — Windows & macOS (`bind()` your driver) | `templates/bolt_on.py` | `qirabot[desktop]` (whole screen, any OS) · `qirabot[airtest]` (Windows only, one window) | Copy the template, fill in the `TODO`s (start URL / app package or bundle id, and the task), then run it with **the interpreter preflight echoed** (its absolute path), not a bare `python`. iOS has two starters; pick by what the user asked for and what's already running: - **`templates/ios_airtest.py`** — drives WDA over HTTP directly. Pick this when the user says "airtest", when WDA is already up on :8100, or when they want the minimal-deps path (no `appium` server). Real device only. - **`templates/ios_appium.py`** — Appium XCUITest. Pick this when the user says "appium", when targeting a simulator, when you need Appium to build/sign WDA for you, or when the script needs Appium's device APIs (e.g. `driver.start_recording_screen`). When the user doesn't say either, default to `ios_airtest.py` if `curl http://127.0.0.1:8100/status` returns ready, else `ios_appium.py`. `templates/bolt_on.py` shows the generic bind-an-existing-driver pattern with Selenium as the runnable example plus pyautogui (whole-screen desktop, any OS) and Airtest (window-scoped Windows desktop) variants in comments; see `references/REFERENCE.md` for the full per-platform action matrix and `bind()` details. ## Step 2 — Hand the task to qirabot (default), drop to primitives only to optimize **Default: give the whole task to `bot.ai`.** ```python from qirabot import StepResult def on_step(step: StepResult) -> None: # live trace -> stdout (see below) label = "done" if step.finished else step.action_type print(f" step {step.step}: {label} {step.params} — {step.decision}") result = bot.ai(target, "Add the cheapest item to the cart and check out", max_steps=15, on_step=on_step) print(result.success, result.output) ``` `bot.ai` offloads the perceive → decide → act loop to qirabot, which manages its own step history and self-heals when a step misfires. **Keep the task string a concise goal, not a step-by-step script.** `bot.ai` is smart enough to plan its own clicks — over-specifying ("click Search, then type X, then click the first result, then…") fights the model, locks in a brittle path, and burns extra steps. Write what success looks like, not how to get there. Good: `"Add the cheapest in-stock item to the cart and check out"`. Bad: a 6-step click-by-click recipe. The examples here pass the target explicitly (`bot.ai(target, ...)`). **If you `bind()` a stable target first** — as the `android.py`, `ios_airtest.py`, `ios_appium.py`, and `bolt_on.py` templates do — drop the leading arg: `bot.ai("...")`, `bot.click("...")`. (Keep the explicit form for Playwright so new-tab follows stay visible — see `references/REFERENCE.md`.) **Always pass `on_step`.** Until it returns, `bot.ai` is a black box — `result` only lands at the end. `on_step` fires after every step and prints the model's running `decision` + action to stdout, which is your one live window into the run: a stuck, looping, or failed run becomes debuggable straight from the console, without opening the HTML report. (`StepResult` also carries `.output` and token/duration counts — see `references/REFERENCE.md`.) **Drop to the per-step primitives only as a deliberate optimization** — when you want strict, reproducible determinism, or you're codifying a stable flow to run repeatedly (e.g. a CI regression check). They cost less per action and are reproducible, but are brittle to UI changes: ```python bot.click(target, "Login button") bot.type_text(target, "Email field", "a@b.com", press_enter=True) bot.double_click(target, "the file name to rename") # double-click bot.long_press(target, "the message bubble", duration=2.5) # mobile: context menu bot.key_down(target, "w"); bot.key_up(target, "w") # desktop: hold/release a key (pair them) text = bot.extract(target, "the displayed account balance") # read one thing bot.wait_for(target, "the dashboard finished loading") # gate, raises on timeout ``` `extract()` reads values off the screen; `verify()` returns a bool the script can branch on. See Step 3 for when to use which after `bot.ai`. See `references/REFERENCE.md` for the full API: constructor options, `bind()`, navigation/scroll/keys, the per-platform action matrix, and errors. ## Step 3 — Check the result by who consumes it Every run writes a self-contained HTML report with per-step screenshots to `./qira_runs///report.html` (unless `report=False`). `bot.ai` leaves three signals after it returns — pick by who acts on them: - **A human/agent will read the result** → open the report and look at the step screenshots. Do NOT trust `result.success`: it's the same model that just acted, reporting on itself, and can claim victory after clicking the wrong button. - **The script must branch on the outcome** (CI gate, `if logged_in skip login`, conditional flow) → `bot.verify(target, "...")`. Independent vision call, returns `bool`, costs one AI call. The bool must drive something — otherwise it's a billed call whose result goes nowhere, and the screenshot already tells the human reviewer the same thing for free. - **The script needs to read a value** (price, username, status) → `bot.extract(target, "...")`. Beware ambiguous locates: `extract("the logged-in username")` can grab a rotating search-box hint instead — scope the phrase and cross-check against the screenshot. When a run fails (`result.success=False`, or the screenshot looks wrong), the **stdout `on_step` trace is the fastest entry point** — find the step where the model started looping or chose a wrong action, then jump to that step's screenshot in the report. **Embed a screen video in the report.** The report auto-discovers a file named `recording.mp4` in `bot.report_dir` and embeds it as a `