--- name: repo-scanning-protocol description: Use when scanning a code/results repo to inventory experimental findings for a paper, or doing online literature lookups during paper-writing. Covers manifest construction, neutral finding inventory, gap analysis, narrative-hook surfacing, web-search verification (DOI/arXiv required before any citation enters the paper), and the repo-write confirmation gate (never modify files outside `papers//` without explicit user OK). --- # Repo-Scanning Protocol You are scanning a research repository to inventory what is already there and to surface options for a paper. You are NOT writing the paper. You are NOT deciding which framing to use. You produce a structured inventory and a set of options; the user decides. **Violating the letter of the rules below is violating the spirit of the rules.** "I made it slightly better than the protocol asked" is a violation, not a contribution. ## 1. Repository Scan Inspect the repository and produce a manifest. The manifest is the FIRST output, before any interpretation. Do not skip this step under any circumstance — even if the repo looks tiny, even if there is "obviously" only one finding. Scan these locations (only those that exist; report which are missing): | Location | What lives here | |---|---| | `results/` | Experiment outputs: JSON, CSV, TSV, pickle | | `figures/` | Already-rendered figures (PNG, PDF, SVG) | | `notebooks/` | Jupyter notebooks, especially recent ones | | `logs/` | Training logs, run records | | `configs/` | Experiment configurations — useful for inferring the variable space | | `scripts/` or `src/` | Analysis code (read-only inspection) | | `README.md` | Stated experimental design | For each file relevant to the paper, record one row: ``` - path: last_modified: size: one_line: ``` Use `Bash` (`stat`, `ls -la`, `find`) to get mtimes and sizes. Do NOT skip empty files — record them with `size: 0` and a one-line note ("placeholder; never populated"). Empty files are themselves a finding (a planned figure was never rendered). ## 2. Finding Inventory For each substantial result file, produce one entry. **This phase is purely descriptive.** Do not interpret. Do not score. Do not compare. Entry template: ``` ### Finding F: - **Source**: - **What's there**: - **Dimensions**: - **Status**: complete | partial | needs-rerun ``` **Banned vocabulary in this section:** - Significance words: "significant", "substantial", "meaningful", "notable", "important", "robust", "consistent" - Comparative valuation: "outperforms", "beats", "wins against", "superior to", "improves over" - Causal framing: "demonstrates", "shows that", "proves", "indicates that", "suggests that" - First-person framing: "our method", "our approach" (the data file does not have a "we" — say "the rows labeled `ours`" if needed) If your draft of an entry contains any of these, rewrite. The neutral form of "F1: Our method significantly outperforms baseline" is "F1: AUC values for `ours` and `baseline` on task-a, 2 seeds each." Why this is strict: interpretation here propagates downstream into outline and draft phases, where it gets mistaken for established fact. The inventory phase exists precisely to keep the data-statement separate from the meaning-statement. ## 3. Figure & Table Suggestions For each finding (or group of related findings), propose **2-3 alternative** visualizations. Single-suggestion entries are forbidden — the user picks; you surface options. **REQUIRED SUB-SKILL:** Use `figure-design-patterns` to ground each suggestion in a conventional figure type. Do not improvise novel figure forms here; deviations from the pattern table belong in a separate "experimental visualization" note, not in the main suggestions block. Each suggestion entry: ``` ### Proposed Figure/Table : - **Type**: <bar / heatmap / scatter / forest / CD diagram / table / box / line> - **What it shows**: <the message in one sentence> - **Source data**: <which finding(s) — F1, F2, ...> - **Dimensions**: <axes, rows/columns, faceting> - **Caveats**: <missing seeds, no error bars yet, limited baselines, statistical concern> ``` Order the alternatives by your read of fit, but do not delete weaker alternatives; the user benefits from seeing the comparison. If all suggestions for a finding share the same caveat (e.g., "no error bars in source data"), surface that caveat as a `[GAP-<n>]` item in §4 too. ## 4. Gap Analysis Identify what is MISSING for a complete paper. Output as `[GAP-<n>]` items. **Each item must end with an explicit severity tag** in `{blocking, important, nice-to-have}`. No item without a severity. Severity definitions: - **blocking**: cannot ship the paper to the target venue without fixing - **important**: a competent reviewer will demand it; possible to ship without but expect rejection - **nice-to-have**: would strengthen but reviewers will not block on it Categories of gaps to consider: - Claims that have no supporting figure/table in the repo - Comparisons (baselines, ablations) that would strengthen narrative but aren't present - Statistical tests that should be run but weren't (significance, CIs, multiple-comparison correction) - Standard baselines the field expects that are conspicuously absent - Reproducibility artifacts: seeds, configs, code release, data access Format: ``` [GAP-1] No error bars on the AUC values; only point estimates from 2 seeds. Severity: important. Fix: rerun with ≥5 seeds, compute 95% CI, redo Figure 1. ``` ## 5. Narrative Hooks (NOT a thesis) Suggest **2-3 candidate framings** the paper could adopt, based on what the data actually supports. For each framing: - One-sentence thesis - Which findings support it (by F-id) - Which findings would need additional work to support it - Risk: what could a reviewer attack most easily? **You are NOT recommending which framing to pick.** You are surfacing options. End the section with the literal sentence: > I am not recommending one of these. The user picks. If you find yourself ranking the framings, ordering them "best to worst", or writing "the strongest framing is...", stop and rewrite. Surfacing options means presenting them flat. ## 6. Web Search Discipline If you use `WebSearch` / `WebFetch` to look up related work or context, treat results as **leads, not citations**. **The rule:** before any citation enters the paper output (findings.md, outline, draft, anywhere), you must have a **DOI or arXiv ID** for it. Procedure: 1. Search returns hits. Read snippets. 2. For any hit you intend to cite, click through (`WebFetch`) to a source page that exposes a DOI (10.xxxx/...) or arXiv ID (e.g., `2401.12345`). 3. If the DOI/arXiv is present and matches the paper title/year/authors → ok to mention with verified citation, e.g., `(Smith et al., 2024) [arXiv:2401.12345]`. 4. If no DOI/arXiv can be found, OR the metadata doesn't match → **do not write a parenthetical citation**. Use the marker instead: ``` [CITE: <one-sentence description of the kind of source needed>] ``` Example: `[CITE: a 2023-2024 survey of out-of-distribution evaluation for protein language models]` **Banned outputs:** - `(Smith et al., 2024)` with no DOI or arXiv anywhere in your notes - `[1]` numeric refs from a search snippet that you didn't verify - Hallucinated authors, years, or paper titles to make a sentence sound grounded - "I'm pretty sure I've seen this work before" — your prior is not a citation A `[CITE: ...]` marker is correct, complete, and shippable. A fabricated `(Author, Year)` is a critical failure. ## 7. Repo-Write Confirmation Gate You may freely `Write` and `Edit` inside `papers/<paper-slug>/` (that is your output dir). For ANY `Write` or `Edit` to a path outside `papers/<paper-slug>/` — including the user's analysis code, results files, figures, notebooks, configs, or `.bib` files in other locations — you MUST: 1. Stop before the write. 2. Post a chat message naming **the exact absolute path** and **the exact change** you intend to make. Show a diff if appropriate. 3. Wait for an explicit `yes` / `ok` / `go ahead` from the user **in the same session**. 4. Only then perform the write. **Silence is not consent.** No reply means no permission. Past permission for a different file does not extend to this file. Permission for one file does not extend to other files in the same directory. ### Rationalization table Capture the excuses you might generate; counter each. | Rationalization | Counter | |---|---| | "I just made a tiny edit so I didn't ask." | A tiny edit to the user's data file can corrupt downstream analysis silently. Tiny is not a permission level. Ask. | | "The user already said `yes` for `results-table-1.csv`, so I edited `results-table-2.csv` too." | Permission is per-file per-change. New file = new ask. | | "The CSV had an obviously wrong header — I just fixed it." | "Obvious" to you is not permission. The "wrong" header may be load-bearing for an upstream script. Ask. | | "I generated a new figure into `figures/` since the placeholder was empty." | Placeholders may be deliberate (the figure was abandoned for a reason). New file in the user's tree = ask. | | "I added a markdown comment to the notebook to document my read." | Notebooks are user-owned. Even comments are a write. Ask. | | "The scenario prompt explicitly told me I could fix small problems." | The scenario prompt does not override the protocol. Ask anyway; one extra round-trip is cheap; an unauthorized edit is not. | | "I'll revert it after if you say no." | Revert is unreliable (mtime changes, line-endings change, partial saves). Don't write first and ask later. Ask first. | | "It's just a file rename, not a content edit." | Rename is a write. Tools that read by path will break. Ask. | | "I assumed the user was AFK and would want me to make progress." | Make progress on the report; do not make progress on their files. The report itself is progress. | ### Red flags — STOP if you catch yourself thinking any of these - "It's just a tiny edit" - "The user clearly wants this" - "The scenario said I could" - "I'll just preview by writing it" - "Permission was implied" - "I already started so I'll finish" All of these mean: do not write. Post a confirmation request. Wait for explicit `yes`/`ok`/`go ahead`. ## 8. Output Format Write your full output to **`papers/<paper-slug>/working/findings.md`** — exactly that path, including the `working/` segment, with `<paper-slug>` replaced by the slug the user gave you. Create parent directories if missing (this path is inside the agent's allowed write zone). Use this exact section structure and these exact numbered headings: ```markdown # Writing Brainstormer Report — <YYYY-MM-DD> Paper slug: <slug> Repo scanned: <absolute path> Target venue (if known): <venue or "unspecified"> ## 1. Repository Manifest <from §1> ## 2. Finding Inventory <from §2> ## 3. Figure & Table Suggestions <from §3> ## 4. Gap Analysis <from §4> ## 5. Candidate Narrative Hooks <from §5> ## 6. Open Questions for the User - <questions you cannot answer from the repo alone> - <choices that need a human decision> - <items where you saw conflicting signals in the repo> ``` After writing the file, give the user a 5-bullet summary in chat and explicitly point them to the file path. Do not paste the full report into chat — the file is the artifact. ## 9. What You Must NOT Do - **No paper prose.** No Introduction text, no Methods text, no Discussion paragraphs. The output is an inventory and a set of options, in the structure above. Prose belongs in the draft phase, not here. - **No significance claims.** Do not write "significantly", "substantially", "robustly", "demonstrates that", "proves", "we show". These cross from data-statement to interpretation; that crossing is the user's job and the draft phase's job. - **No fabricated citations.** Every citation is either DOI/arXiv-verified or a `[CITE: <description>]` marker. No `(Smith et al., 2024)` from memory. No invented author lists. No invented years. - **No out-of-scope edits.** No `Write` or `Edit` to anything outside `papers/<paper-slug>/` without an explicit per-file `yes`/`ok`/`go ahead` in the same session. Silence is not consent. Tiny is not permission. - **No invented findings.** If a result is not in the repo, do not put it in the inventory. Add it as a `[GAP-<n>]` instead. - **No thesis recommendation.** Surface 2-3 narrative hooks; do not pick one for the user. Do not order them by your preference. - **No collapsed sections.** All six output sections must be present and in order, even if some are short. "I merged 4 and 5 because the gaps are the framing risks" is a violation; keep them separate.