--- name: wiki-ingest description: "Ingest sources into the Obsidian wiki vault. Reads a source, extracts entities and concepts, creates or updates wiki pages, cross-references, and logs the operation. Supports files, URLs, and batch mode. Triggers on: ingest, process this source, add this to the wiki, read and file this, batch ingest, ingest all of these, ingest this url." --- # wiki-ingest: Source Ingestion Read the source. Write the wiki. Cross-reference everything. A single source typically touches 8-15 wiki pages. **Syntax standard**: Write all Obsidian Markdown using proper Obsidian Flavored Markdown. Wikilinks as `[[Note Name]]`, callouts as `> [!type] Title`, embeds as `![[file]]`, properties as YAML frontmatter. If the kepano/obsidian-skills plugin is installed, prefer its canonical obsidian-markdown skill for Obsidian syntax reference. Otherwise, follow the guidance in this skill. --- ## Delta Tracking Before ingesting any file, check `.raw/.manifest.json` to avoid re-processing unchanged sources. ```bash # Check if manifest exists [ -f .raw/.manifest.json ] && echo "exists" || echo "no manifest yet" ``` **Manifest format** (create if missing): ```json { "sources": { ".raw/articles/article-slug-2026-04-08.md": { "hash": "abc123", "ingested_at": "2026-04-08", "pages_created": ["wiki/sources/article-slug.md", "wiki/entities/Person.md"], "pages_updated": ["wiki/index.md"] } } } ``` **Before ingesting a file:** 1. Compute a hash: `md5sum [file] | cut -d' ' -f1` (or `sha256sum` on Linux). 2. Check if the path exists in `.manifest.json` with the same hash. 3. If hash matches, skip. Report: "Already ingested (unchanged). Use `force` to re-ingest." 4. If missing or hash differs, proceed with ingest. **After ingesting a file:** 1. Record `{hash, ingested_at, pages_created, pages_updated}` in `.manifest.json`. 2. Write the updated manifest back. Skip delta checking if the user says "force ingest" or "re-ingest". --- ## URL Ingestion Trigger: user passes a URL starting with `https://`. Steps: 1. **Fetch** the page using WebFetch. 2. **Clean** (optional): if `defuddle` is available (`which defuddle 2>/dev/null`), run `defuddle [url]` to strip ads, nav, and clutter. Typically saves 40-60% tokens. Fall back to raw WebFetch output if not installed. 3. **Derive slug** from the URL path (last segment, lowercased, spaces→hyphens, strip query strings). 4. **Save** to `.raw/articles/[slug]-[YYYY-MM-DD].md` with a frontmatter header: ```markdown --- source_url: [url] fetched: [YYYY-MM-DD] --- ``` 5. Proceed with **Single Source Ingest** starting at step 2 (file is now in `.raw/`). --- ## Image / Vision Ingestion Trigger: user passes an image file path (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.svg`, `.avif`). Steps: 1. **Read** the image file using the Read tool. Claude can process images natively. 2. **Describe** the image contents: extract all text (OCR), identify key concepts, entities, diagrams, and data visible in the image. 3. **Save** the description to `.raw/images/[slug]-[YYYY-MM-DD].md`: ```markdown --- source_type: image original_file: [original path] fetched: YYYY-MM-DD --- # Image: [slug] [Full description of image contents, transcribed text, entities visible, etc.] ``` 4. Copy the image to `_attachments/images/[slug].[ext]` if it's not already in the vault. 5. Proceed with **Single Source Ingest** on the saved description file. Use cases: whiteboard photos, screenshots, diagrams, infographics, document scans. --- ## Single Source Ingest Trigger: user drops a file into `.raw/` or pastes content. Steps: 1. **Read** the source completely. Do not skim. 2. **Discuss** key takeaways with the user. Ask: "What should I emphasize? How granular?" Skip this if the user says "just ingest it." 3. **Create** source summary in `wiki/sources/`. Use the source frontmatter schema from `references/frontmatter.md`. Assign an address per the **Address Assignment** section below. 4. **Create or update** entity pages for every person, org, product, and repo mentioned. One page per entity. Assign addresses to new entity pages. 5. **Create or update** concept pages for significant ideas and frameworks. Assign addresses to new concept pages. 6. **Update** relevant domain page(s) and their `_index.md` sub-indexes. 7. **Update** `wiki/overview.md` if the big picture changed. 8. **Update** `wiki/index.md`. Add entries for all new pages. 9. **Update** `wiki/hot.md` with this ingest's context. 10. **Append** to `wiki/log.md` (new entries at the TOP): ```markdown ## [YYYY-MM-DD] ingest | Source Title - Source: `.raw/articles/filename.md` - Summary: [[Source Title]] - Pages created: [[Page 1]], [[Page 2]] - Pages updated: [[Page 3]], [[Page 4]] - Key insight: One sentence on what is new. ``` 11. **Check for contradictions.** If new info conflicts with existing pages, add `> [!contradiction]` callouts on both pages. --- ## Batch Ingest Trigger: user drops multiple files or says "ingest all of these." Steps: 1. List all files to process. Confirm with user before starting. 2. Process each source following the single ingest flow. Defer cross-referencing between sources until step 3. 3. After all sources: do a cross-reference pass. Look for connections between the newly ingested sources. 4. Update index, hot cache, and log once at the end (not per-source). 5. Report: "Processed N sources. Created X pages, updated Y pages. Here are the key connections I found." Batch ingest is less interactive. For 30+ sources, expect significant processing time. Check in with the user after every 10 sources. --- ## Context Window Discipline Token budget matters. Follow these rules during ingest: - Read `wiki/hot.md` first. If it contains the relevant context, don't re-read full pages. - Read `wiki/index.md` to find existing pages before creating new ones. - Read only 3-5 existing pages per ingest. If you need 10+, you are reading too broadly. - Use PATCH for surgical edits. Never re-read an entire file just to update one field. - Keep wiki pages short. 100-300 lines max. If a page grows beyond 300 lines, split it. - Use search (`/search/simple/`) to find specific content without reading full pages. --- ## Contradictions > [!note] Custom callout dependency > The `[!contradiction]` callout type used below is a **custom callout** defined in `.obsidian/snippets/vault-colors.css` (auto-installed by `/wiki` scaffold). It renders with reddish-brown styling and an alert-triangle icon when the snippet is enabled. If the snippet is missing, Obsidian falls back to default callout styling, so the page still works without the visual flourish. See [[skills/wiki/references/css-snippets.md]] for the four custom callouts (`contradiction`, `gap`, `key-insight`, `stale`). When new info contradicts an existing wiki page: On the existing page, add: ```markdown > [!contradiction] Conflict with [[New Source]] > [[Existing Page]] claims X. [[New Source]] says Y. > Needs resolution. Check dates, context, and primary sources. ``` On the new source summary, reference it: ```markdown > [!contradiction] Contradicts [[Existing Page]] > This source says Y, but existing wiki says X. See [[Existing Page]] for details. ``` Do not silently overwrite old claims. Flag and let the user decide. --- ## What Not to Do - **Source files under `.raw/` are immutable.** Do not modify the files that users drop there (articles, transcripts, images). The `.raw/.manifest.json` delta tracker and its `address_map` (DragonScale Mechanism 2) are the only files under `.raw/` that `wiki-ingest` itself maintains. Treat every other file under `.raw/` as read-only source content. - Do not create duplicate pages. Always check the index and search before creating. - Do not skip the log entry. Every ingest must be recorded. - Do not skip the hot cache update. It is what keeps future sessions fast. --- ## Address Assignment (DragonScale Mechanism 2 MVP) **Opt-in feature**. DragonScale address assignment runs only if `scripts/allocate-address.sh` is present AND `.vault-meta/` exists. Otherwise, skip this entire section and proceed with ingest normally. **Feature detection (run at start of every ingest)**: ```bash if [ -x ./scripts/allocate-address.sh ] && [ -d ./.vault-meta ]; then DRAGONSCALE_ADDRESSES=1 else DRAGONSCALE_ADDRESSES=0 fi ``` When `DRAGONSCALE_ADDRESSES=0`, pages are created without an `address:` frontmatter field, and `wiki-lint`'s Address Validation section is skipped entirely (missing addresses are not flagged in any severity). This preserves default plugin behavior for vaults that have not adopted DragonScale. When `DRAGONSCALE_ADDRESSES=1`, proceed with the rest of this section. --- Every **newly created non-meta wiki page** gets a stable address in its frontmatter: ```yaml address: c-000042 ``` Format: `c-<6-digit-counter>`. The `c-` prefix stands for "creation-order counter." Zero-padded. Rollout baseline: **2026-04-23** (Phase 2 ship date). Pages with `created:` >= this date are post-rollout and MUST have an address (unless excluded below). Pages with `created:` earlier are legacy-exempt until a deliberate backfill pass assigns `l-NNNNNN` addresses. ### Required tool: `scripts/allocate-address.sh` Address allocation is delegated to an atomic Bash helper. The helper uses `flock` on `.vault-meta/.address.lock` to prevent read-use-increment races and recovers the counter by scanning existing frontmatter if the counter file is missing. ```bash ADDR=$(./scripts/allocate-address.sh) # ADDR is now e.g. "c-000042"; counter is already incremented ``` **CRITICAL**: never use the Write or Edit tool on `.vault-meta/address-counter.txt`. That would fire the PostToolUse hook, which runs `git add wiki/ .raw/` and can accidentally commit unrelated pending wiki changes under a generic message. Counter mutation is **only** permitted through the helper script (Bash tool). ### Helper modes - `./scripts/allocate-address.sh` — atomically reserves and returns the next address. - `./scripts/allocate-address.sh --peek` — prints the next value without reserving (safe, read-only). - `./scripts/allocate-address.sh --rebuild` — recomputes the counter from the highest observed `c-NNNNNN` in existing frontmatter. Never resets to 1 silently if pages already have addresses. Run this if the counter file is suspected corrupt. ### Assignment procedure (per new page) 1. Before writing a new non-meta page, call `./scripts/allocate-address.sh` and capture the output. 2. Include `address: c-XXXXXX` in the page's frontmatter. 3. Record the path-to-address mapping in `.raw/.manifest.json` under a new top-level key `address_map` (see schema below). ### `address_map` in `.raw/.manifest.json` ```json { "sources": { ... }, "address_map": { "wiki/concepts/Example.md": "c-000042", "wiki/entities/Another.md": "c-000043" } } ``` On re-ingest of the same source (whether by `--force` or a changed hash), always consult `address_map` first. If the target page path has a prior address, REUSE it. Do not allocate a new one. On a page rename, the skill must update the `address_map` key (old path -> new path) while preserving the address value. ### Exclusions (do NOT assign an address to) - Meta files: `_index.md`, `index.md`, `log.md`, `hot.md`, `overview.md`, `dashboard.md`, `dashboard.base`, `Wiki Map.md`, `getting-started.md`. - Fold pages under `wiki/folds/` (they use their own deterministic `fold_id`). - Pre-rollout legacy pages (`created:` < 2026-04-23). Legacy pages get `l-NNNNNN` addresses only via a deliberate backfill operation. ### Idempotency rules - If a page being (re)written already has an `address:` field in its current content, REUSE it. Do not allocate a new one. - If a source is re-ingested and `address_map` has a mapping for the target path, reuse that mapping. - If the source has been ingested before AND the target page has no address AND the page `created:` date is post-rollout, allocate an address and record it. This covers the case where an older ingest produced a page before Phase 2 rollout; the rollout cutoff still applies (pages dated pre-2026-04-23 stay legacy). ### Concurrency policy - **Single-writer only** in Phase 2. Do not run parallel ingests from multiple Claude sessions or sub-agents that assign addresses. The `flock` in the helper prevents counter corruption but does not serialize page writes themselves. - Sub-agents (codex, general-purpose) that are dispatched for research or review MUST NOT call the allocator. They are read-only in this respect. - Multi-writer support is a deferred feature. ### Batch ingest Assign addresses sequentially during single-source-ingest for each source. Do not pre-reserve a block of counter values. The helper is cheap (one lock, one integer read/write).