--- name: research-exp-live description: "Use when launching, monitoring, resuming, or stopping a long-running research Experiment with structured run evidence and scheduled live checks." --- # research-exp-live ## Purpose Use this skill for a long-running command that belongs to an existing Package and Experiment. It authorizes the Run through management state, stores producer evidence in one Run directory, and keeps monitoring decisions tied to measured status. This skill does not create a Package or Experiment, decide research scope, or treat a browser page as state. ## Storage and authority Resolve the managed root once: ```bash export RESEARCH_ROOT="${RESEARCH_ROOT:-.research}" ``` `--research-root` overrides the environment variable. If neither is set, the default is `.research`. With the default root, the three managed areas are `.research/state/`, `.research/experiments/`, and `.research/interface/`. | Data | Location | Authority | | --- | --- | --- | | Package, Experiment, Run, and allocation lifecycle | `$RESEARCH_ROOT/state/events.jsonl` | Management authority | | Current folded state | `$RESEARCH_ROOT/state/current.json` | Rebuildable state projection | | Run command and frozen context | `$RESEARCH_ROOT/experiments////run.json` and `context.json` | Immutable Run envelope | | Live status and raw evidence | The same Run directory | Producer-owned runtime evidence | | Human pages | `$RESEARCH_ROOT/interface/` | Read-only generated projection | | Interface server process metadata | `$XDG_RUNTIME_DIR/trustworthy-research//` | Ephemeral local runtime | When `XDG_RUNTIME_DIR` is unset, `ResearchPaths` uses its per-user temporary runtime fallback. Server metadata never belongs in `$RESEARCH_ROOT/state/` or `$RESEARCH_ROOT/interface/`. The interface can be absent, stale, or rebuilt while an Experiment is running. Interface health never authorizes a launch and never proves completion. ## Preconditions Before launch, confirm through state queries that: - the Package lifecycle is `ACTIVE` and its phase is `READY_TO_LAUNCH`; - the selected Experiment belongs to that Package and has status `READY`; - `Experiment.spec` contains `purpose`, `config_ref`, `gate`, and `control_mode`; - a user `LAUNCH_ACK` or `READY_TO_LAUNCH_ACK` Decision exists; - a requested GPU allocation is open and bound to the same Package and Experiment. The launcher checks these conditions again under the management-state lock. Do not bypass a rejected launch by writing a Run directory yourself. ## Launch Use the canonical launcher: ```bash python3 -m lib.experiments.launch \ --workspace . \ --research-root "$RESEARCH_ROOT" \ --pkg \ --exp \ --tmux-session \ --heartbeat-timeout 600 \ -- bash scripts/run_experiment.sh ``` The default transport is a named tmux session. Use `--foreground` for a short command. Useful optional flags include: - `--metrics-regex` for named metric groups; - `--total-steps` for progress normalization; - `--wandb-run-id` and `--tensorboard-logdir` for external telemetry identifiers; - `--expected-duration minutes|hours|days` for scheduling; - `--gpu-sample` for GPU observations; - `--server` and `--alloc` for a resource binding; - `--retry-of ` for a replacement attempt. External telemetry adds observations. It does not replace the Run directory or management callbacks. The launcher writes: ```text $RESEARCH_ROOT/experiments//// ├── run.json ├── context.json ├── status.json ├── events.jsonl ├── metrics.jsonl ├── result.json └── log.txt ``` `run.json` binds the command, context hash, selected Experiment, resource, and transport. `context.json` freezes the state snapshot used at authorization. ## Optional human interface Start or reuse the read-only interface server only when a human view is useful: ```bash python3 -m lib.interface.serve \ --workspace . \ --research-root "$RESEARCH_ROOT" \ ensure --json ``` The server serves `$RESEARCH_ROOT/interface/` and exposes narrow read-only APIs over state and Run evidence. Its `dashboard_server.json` and log live under the XDG runtime directory. A failed `ensure` call is interface debt, not a Run failure. ## Startup health gate Check the new Run within 120 seconds unless the command has a documented slow start. Read `status.json`. Startup is confirmed only when: - the lifecycle is still open or a terminal status is mechanically recorded; - `first_output_at` is set; - `health.state` is `OK` or `WARN`. Do not report a healthy launch before these facts exist. If the process exits or `health.state=ERROR`, inspect a bounded log tail: ```bash python3 -m lib.experiments.report \ --workspace . \ --research-root "$RESEARCH_ROOT" \ --run "$RESEARCH_ROOT/experiments///" \ --tail 50 ``` Route the verified failure to repair, implementation review, a user decision, or a recorded blocker. ## Monitoring cadence For the first 30 minutes, track the Run every 5 minutes. If a failed branch is replaced, restart that 30-minute window from the replacement launch. After the initial window, use the best available scheduling evidence: 1. measured progress and stable throughput from `status.json`; 2. recent progress changes and output heartbeat; 3. `expected_duration` from `run.json`; 4. unknown duration. | Remaining duration | Next check | | --- | --- | | up to 15 minutes | 2 to 5 minutes | | up to 1 hour | about 10 minutes | | up to 6 hours | 15 to 30 minutes | | over 6 hours | 30 to 60 minutes | | unknown | 10 minutes | No open Run goes unchecked for more than 60 minutes. Check immediately when status becomes `STALE`, health becomes `ERROR`, anomaly count increases, or throughput drops sharply. At every check: 1. Read `status.json` for each management-open Run. Use `log.txt` only to diagnose a problem. 2. Record the live-check observation through a `research-op` command envelope. 3. Record `Next Check` and arm re-entry before that time. 4. Reconcile missing management callbacks. 5. Run the Package artifact scan once. Do not create a second polling loop. ```bash python3 -m lib.experiments.reconcile \ --workspace . \ --research-root "$RESEARCH_ROOT" python3 skills/research-op/scripts/research_op.py \ --workspace . \ --research-root "$RESEARCH_ROOT" \ --pkg \ --op scan-events \ --payload '{}' ``` The observation fields come from `status.json`: `status`, `last_output_at`, `progress`, `latest_metrics`, `resource`, and health. The agent supplies the action and next-check time. Do not edit state or interface files to record a check. ## Liveness and failure Liveness comes from the child `pid`, `harvester_pid`, `last_output_at`, and `heartbeat_timeout`. `status.json` can become `STALE` even when the command is quiet. Verify the process before deciding whether to wait, stop, or replace it. Canonical terminal statuses are: - `COMPLETED` - `FAILED` - `HALTED` - `SKIPPED` Completion requires a terminal `status.json` with `exit_code` and `ended_at`, followed by a matching management callback. If the callback is missing, run the reconciler. Elapsed time, an expired estimate, or a quiet log is not completion evidence. After terminal status: 1. read `result.json` and its EvidenceRefs; 2. compare measured results with the immutable Experiment gate; 3. record result and Experiment status through `research-op`; 4. release any open resource allocation; 5. run reconciliation and artifact scan again. ## Open-run stop gate Before ending a session, list management-open Runs: ```bash python3 -m lib.experiments.report \ --workspace . \ --research-root "$RESEARCH_ROOT" \ --open ``` Every listed Run needs a scheduled next check. If a Run is already terminal on disk but still appears open, reconcile it before stopping. A healthy interface server does not clear this gate. ## Worked example ```text status=RUNNING health=OK progress=1200/50000 latest_metrics={"loss":0.41} Decision: continue Next Check: 5 minutes, still inside the initial 30-minute tracker window Management: live-check command accepted Reconcile: no missing callback Artifact scan: no new event ``` Failure: ```text status=FAILED health=ERROR exit_code=1 first_output_at=set Decision: startup health refused Evidence: bounded report tail shows ImportError Route: FIX_IMPLEMENTATION Replacement: launch with --retry-of and restart the 30-minute tracker ``` ## References - [status contract](references/status-contract.md) - [telemetry sources](references/telemetry-sources.md) - [live page contract](references/live-page-contract.md)