--- name: update-references description: >- Find, download, read, and summarize research papers related to the work in the current directory. Searches arXiv, OpenAlex, Semantic Scholar, and Google Scholar, with a hierarchical mode (hubs, surveys, citation snowball). Maintains a CSL-YAML reference database (references.yaml) compatible with pandoc's --bibliography flag, so the same file serves both the skill and document builds. Triggers: update references, refresh the bibliography, find papers, related work, literature review, what's been published on, recent papers, cite sources for, background reading, summarize this paper, search scholar. --- # Update references (research paper search) This skill turns "find me what's been written about this" into a small, repeatable pipeline: read what the user is working on, search arXiv (and, for off-arXiv work, Semantic Scholar or Google Scholar) for related papers, download the relevant ones, read them, write a summary per paper, and keep a Citation Style Language (CSL) YAML database so work is never repeated and versions stay current. The standing interests are LLMs, AI agents, finite state machines, and declarative agent patterns — the territory of spindle (github.com/petar-djukic/spindle), a Go state-machine engine for agentic loops. Lean toward those topics, but always sharpen the search using whatever the current working directory is actually about. ## Where things live This skill is context-aware. It runs from the directory the user is working in, and it stores its outputs there — not in a fixed global location. Resolve paths like this: - **Database:** `references.yaml` under the current working directory, unless an existing one is already nearby (look for a `references.yaml` at or above the working directory and reuse it). Pass the chosen path as `--db` to every script call so search, fetch, and record all agree. If only a legacy `arxiv/papers.yaml` exists, the scripts read it transparently and convert on next write. - **PDFs:** `/pdfs/` (transient — they exist to be read; they don't need to be committed). The script resolves this **relative to the database**, not the current directory, so PDFs land next to the db even when run from elsewhere. - **Papers (markdown):** `/papers/` — `fetch` converts each PDF to markdown automatically (best-effort via `pymupdf4llm`), preserving headings, tables, and math. Papers are readable in any editor or tool. - **Summaries:** `/summaries/`, one file per paper. The PDF, markdown, and summary of a paper share one human-friendly stem: `----` (for example `Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052`). The `-` tail is `arxiv-vN`, `doi-`, or `scholar-`, so any file traces back to its source at a glance. This differs from the pandoc citation key, which stays short — just `-` (see the database format). `repair` renames files created under the older naming scheme. - **Manual-download list:** `/downloads-needed.md` — regenerated by `scholar.py pending`, a checklist of papers that could not be downloaded (paywalled or behind a login) with a landing URL for each. Derived from the database, so it is disposable, not hand-maintained. An entry carrying `download: deferred` is kept off the checklist (it lists compactly at the bottom instead) — set it on metadata-only entries the current work does not cite, and remove it when a citation pass starts citing the entry. If the user names a directory or an existing database, use that instead. Because every output path is derived from `--db`, passing an absolute `--db` keeps all artifacts together regardless of the working directory. ## The database format The database is CSL (Citation Style Language) YAML — by default a bare list with no root key. Each entry has standard CSL fields that pandoc understands (`id`, `type`, `title`, `author`, `container-title`, `URL`, `issued`) plus skill-internal fields (`status`, `version`, `pdf_path`, `arxiv_id`, etc.) that pandoc ignores. This means the file is directly usable as `pandoc --bibliography references.yaml` with no conversion step. A `references:` root key is also accepted, because that is the form pandoc's own documentation shows and what an existing pandoc bibliography usually looks like; so is the skill's earlier `papers:` key. Whichever shape a file arrives in is preserved when the skill writes it back, so pointing the skill at a bibliography you already maintain does not reformat it. Any other shape is refused rather than read as an empty database, and no command will write an empty database over a file that had entries. Every write goes out in `yq`'s normal form: the skill runs `yq -i '.'` over the file after serializing it. PyYAML and yq indent nested sequences differently, and on a bibliography of a few hundred entries that disagreement turns any hand-edit made with yq into a diff the length of the file. One tool decides the format, and it is the one the operator edits with. Without `yq` on PATH the database is still written and still valid, in PyYAML's style, and the skill says so on stderr — the next yq edit will reformat it whole. An entry looks like: ```yaml - id: lee-2026 type: article title: "Meta-Harness: End-to-End Optimization of Model Harnesses" author: - family: Lee given: Yoonho container-title: arXiv preprint arXiv:2603.28052 URL: https://arxiv.org/abs/2603.28052 issued: year: 2026 arxiv_id: "2603.28052" version: 1 status: downloaded pdf_path: pdfs/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.pdf md_path: papers/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md ``` The `id` field is a pandoc citation key (used as `@lee-2026` in markdown), generated from the first author's family name and the year — just `-`, with a letter suffix on collision (`lee-2026a`). It is deliberately short; the richer provenance lives in the file names. The `arxiv_id` field is the base arXiv identifier used for deduplication and version tracking. Entries fetched through OpenAlex additionally carry two marker blocks that later steps use (pandoc ignores them): ```yaml openalex_id: W2158864412 ranking: cited_by: 5671 # citations TO the paper references_count: 52 # citations IN the paper fwci: 102.6 # field-weighted citation impact (1.0 = field average) citation_percentile: 100.0 venue: IEEE Transactions on Software Engineering institutions: [Massachusetts Institute of Technology] institution_types: [education] retrieved: 2026-07-15 # counts age; they are dated discovery: method: survey-references # seed-search | survey-references | forward-citations | author-drill via: "Wang et al. 2024 autonomous-agents survey" ``` `ranking` answers "how important, by whose measure, as of when"; `discovery` answers "how did this paper enter the corpus". Both persist so the reading step can prioritize and the summaries can cite them. ## Running the scripts The Python scripts run in a pixi-managed environment that ships beside the skill: `pixi.toml` and `pixi.lock` sit at the root of the agent directory (the parent of this skill's `skills/` folder). Provision it once per machine — the agent does this on repo open — with the preflight: ```bash /scripts/ensure-env.sh ``` Then run every script through pixi. For brevity the commands below write `$RUN` for the wrapper: ```bash RUN="pixi run --manifest-path /../../pixi.toml python" ``` where `/../../` resolves to the agent directory holding `pixi.toml`. This replaces the older `pip install --user` path — no packages are installed into the ambient interpreter. ## The workflow ### 0. Reconcile the database Run this first, every time the skill is invoked. The goal is a complete database in the correct naming convention before anything else happens: ```bash $RUN /scripts/arxiv.py --db reconcile ``` `reconcile` (the same command as `repair`) walks the whole database and the `pdfs/` directory and makes disk and db agree: - an entry missing its markdown is converted; - pdf/markdown/summary files on the older naming scheme are renamed to the human-friendly stem (paths updated in the db; citation ids left intact); - a PDF in `pdfs/` that no entry references is imported — its metadata is recovered from an arXiv id in the filename (fetched from arXiv), or, failing that, from the PDF's embedded title/author (recorded `status: needs-review` with year `nd`). Anything unrecoverable is listed in `/unregistered-pdfs.md` with a ready-to-run `ingest` command. It is idempotent and prints counts (`converted`, `renamed`, `collisions`, `imported`, `needs_review`, `unregistered`). A non-zero `collisions` means two entries resolved to the same filename — usually a duplicated citation id — so the rename was refused and those files were left on their old names rather than one overwriting the other. Fix the duplicate and re-run. The same count covers a refused import: an orphan that identifies as a paper the database already holds is a duplicate copy, not a new paper, so it is left in `pdfs/` under its own name instead of being renamed over the copy already there. Delete it, or correct the entry it duplicates. To register an unidentified PDF by hand, give `ingest` the metadata and it creates the entry: ```bash $RUN /scripts/scholar.py --db ingest --file pdfs/.pdf \ --title "Exact Title" --authors "Given Family" --year 2024 ``` After reconciling, refresh the Obsidian tags and source links on every summary (safe to run each time — it only changes what is stale): ```bash $RUN /scripts/keywords.py --db tag ``` ### 1. Understand the current work first Before searching, read what's in the working directory — a draft paper, notes, an outline, existing summaries. The search is only as good as its query, and the query should come from the actual problem the user is working on, not just the standing topic list. If the directory is empty or the intent is unclear, ask the user what angle they care about (one question, then proceed). ### 2-3. Search, dedupe, fetch Per-backend mechanics — source order, dedupe rules, and how each fetch path behaves — are in [search-and-fetch.md](./references/search-and-fetch.md). In short: search the hierarchical backends, dedupe against `references.yaml` before fetching anything, fetch the PDF where a direct link exists, and record a `metadata-only` entry where it does not. ### 4. Read and summarize When several papers await reading, use the `ranking` markers to order them: pivotal first (high `fwci`/`cited_by`), and read a survey before the papers discovered through it (`discovery.via` tells you which those are). Read the markdown conversion that `fetch` produced — its path is in the `md_path` field of the fetch output and the db entry (under `/papers/`). If the markdown file is missing (e.g. an older fetch before markdown conversion was available), run `repair` first to regenerate it. Then write a summary file following `references/summary-template.md` exactly. The summary's job is to let the user decide, in two minutes, whether to cite the paper — so the "Relevance to the current work" section carries the weight. Tie findings back to the current draft and, where it fits, to spindle's state-machine / declarative-agent view. Prefer the paper's own numbers over adjectives. An honest "low relevance" beats a stretch. Name the summary with the same stem as the paper's PDF and markdown so its source is obvious: `/summaries/----.md` (the stem is the basename of the entry's `md_path`, with the `summaries/` directory). If you are unsure of the exact stem, run `repair` afterward — it renames any summary to match. ### 5. Record it Close the loop so the database reflects reality: ```bash $RUN /scripts/arxiv.py --db record --id 2310.12345 \ --summary-file summaries/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md \ --topics llm agents fsm declarative-agents \ --relevance "One line on why it matters to this work." ``` This flips the entry to `status: summarized`. The database is now the single source of truth: re-running a search later will mark these papers `known` and skip them. Then apply Obsidian tags and source links to the new summaries: ```bash $RUN /scripts/keywords.py --db tag ``` This walks the db and, for every summary, writes a normalized `tags:` block into the frontmatter — merging the paper's declared keywords (or a frequency fallback when it declares none) with the entry's `topics`, plus a `paper` root tag — and adds a `**Source:**` line linking the local PDF, the converted full text, and the source URL. It also drops a one-time PDF back-link at the top of each converted paper. Tags a human added are preserved; `--dry-run` previews without writing; a second run is a no-op. Because it walks the whole db, it also tags any older summary that predates this step, so a corpus converges. ### 6. Report back Summarize what was found: how many candidates, how many were new vs. already known, which were summarized, and a one-line takeaway per paper with a link to its summary file. Point out the two or three most relevant to the current work. If any papers came back `metadata-only`, run `scholar.py pending` and hand the user `downloads-needed.md` so they can fetch those PDFs and return them via `ingest`. ### Repair / reconcile `repair` and `reconcile` are the same command — the database reconcile run at step 0. Run it any time to bring disk and db back into agreement: ```bash $RUN /scripts/arxiv.py --db reconcile ``` It walks every entry, re-converts any missing `md_path` from the PDF on disk, migrates legacy `text_path` entries, and renames any pdf/markdown/summary still on the older naming scheme to the human-friendly stem (updating `pdf_path`, `md_path`, `summary_file`; citation `id` values left unchanged). It then scans `pdfs/` for files no entry references and imports them (arXiv-id recovery, then embedded-metadata `needs-review`, then the `unregistered-pdfs.md` list). It is idempotent — correctly-named, tracked files are left alone — and no step ever moves a file onto one that already exists: a rename or an import that would replace another paper is refused, counted as a `collision`, and explained on stderr. Prints a JSON summary (`checked`, `converted`, `renamed`, `collisions`, `imported`, `needs_review`, `unregistered`, `skipped`). ## Dependencies The pixi environment (see "Running the scripts") supplies PyYAML (required), `pymupdf4llm` (PDF-to-markdown conversion preserving headings, tables, and math), and `pypdf` (the plain-text fallback). It is provisioned by `ensure-env.sh`; no `pip install` is needed. If the conversion libraries were somehow absent, fetch still downloads the PDF and just skips the conversion. `yq` (the Go implementation, `brew install yq`) normalizes the database on write; it is optional, and its absence costs formatting, not correctness. Everything else is Python stdlib plus the arXiv public API — no key needed. Be a good citizen: the script already retries with backoff; don't hammer the API with huge `--max` values in a tight loop. OpenAlex needs no key — set `OPENALEX_MAILTO` (any email) for the polite pool's higher rate limits. Google Scholar search requires a SerpAPI key (same key as the idea-factory job-search skill). Semantic Scholar needs no key — the public Graph API is open, though shared and rate-limited; an optional `SEMANTIC_SCHOLAR_API_KEY` (or `--api-key`) raises the limit.