--- name: wiki-gen description: > Compile personal data (journals, notes, messages, whatever) into a personal knowledge wiki. Ingest any data format (Day One JSON, Apple Notes, Obsidian, Notion, plain text, iMessage, CSV, email, Twitter), absorb entries into wiki articles, query the wiki, cleanup articles, and expand coverage. Use when the user wants to build a personal knowledge base from raw journal/diary/notes data, compile entries into Wikipedia-style articles, or query their personal history. Activates on mentions of "personal wiki", "knowledge wiki", "wiki-gen", "compile my journal", "ingest my notes", or sub-commands like "wiki ingest", "wiki absorb", "wiki query", "wiki cleanup", "wiki breakdown", "wiki status". license: MIT metadata: source: "https://gist.github.com/farzaa/c35ac0cfbeb957788650e36aabea836d" original_author: "farzaa" compatibility: | Helper scripts under scripts/ require Python 3.10+ and PyYAML. Install dependency with: pip install pyyaml # Personal Knowledge Wiki You are a **writer** compiling a personal knowledge wiki from someone's personal data. Not a filing clerk. A writer. Your job is to read entries, understand what they mean, and write articles that capture understanding. The wiki is a map of a mind. ## Quick Start Put your data files in the project root (or a `data/` subdirectory). Then run one of these sub-commands via natural language or direct invocation: ``` wiki ingest # Convert your data into raw markdown entries wiki absorb all # Compile entries into wiki articles wiki query # Ask questions about the wiki wiki cleanup # Audit and enrich existing articles wiki breakdown # Find and create missing articles wiki status # Show stats ``` When the user says "wiki ingest", "ingest my journal", or similar, jump to the `wiki ingest` section below and execute it. Same for the other commands. ## What This Wiki IS A knowledge base covering one person's entire inner and outer world: projects, people, ideas, taste, influences, emotions, principles, patterns of thinking. Like Wikipedia, but the subject is one life and mind. Every entry must be absorbed somewhere. Nothing gets dropped. But "absorbed" means understood and woven into the wiki's fabric, not mechanically filed into the nearest article. The goal is never to decide "where to put a fact." The goal is to decide what the fact means and how it connects to what is already known. --- ## Directory Structure ``` your-project/ data/ # Your source files (DO NOT MODIFY after ingest) raw/entries/ # One .md per entry (generated by ingest) raw/batches/ # Optional batch manifests for large vaults wiki/ # The compiled knowledge base _index.md # Master index with aliases _backlinks.json # Reverse link index _absorb_log.json # Tracks which entries have been absorbed _uncovered.md # Coverage gaps after absorb/remediate {directories}/ # Emerge from the data. Don't pre-create. ``` You can override the wiki directory with `--dir ` on any command. Each version is its own universe. Never read from or reference another version. --- ## Command: `wiki ingest` Convert source data into individual `.md` files in `raw/entries/`. This step is mechanical. Prefer the bundled `scripts/ingest_obsidian.py` helper over writing custom parsers. For other formats, copy the closest helper and keep ingest behavior consistent across formats. Key rules: - **Script path resolution**: run bundled helpers by relative path first. If resolution fails, use the documented glob fallbacks. Never write a replacement script from scratch. - **C6 Date Extraction**: resolve dates by strict 8-tier priority and validate against the real calendar. - **C7 Standard Exclusions**: skip `.git`, `.obsidian`, `.claude`, `node_modules`, caches, submodules, and empty files. - **Output**: generate idempotent `{date}_{id}.md` files with YAML frontmatter and track `extra.date_source`. See `references/ingest-spec.md` for script execution and path resolution, supported data formats, standard exclusions, output format, date extraction, warning thresholds, and unknown-format handling. --- ## Scale Mode Before running `wiki absorb`, choose the mode based on raw entry count. Small vaults can run single-pass or sectioned. Large vaults should use partitioned parallel processing with batch manifests and targeted remediation before cleanup. Key rules: - `<= 100` entries: single-pass, chronological, checkpoint every 15 entries - `100-500` entries: sectioned, one agent, checkpoint every 30-50 entries or every source section - `500+` entries: partitioned parallel, unique directory ownership, batch manifests, coverage check, remediation, then cleanup and breakdown - Parallel agents may read `_index.md`, but cross-directory writes are forbidden See `references/scale-mode.md` for the full mode table, partitioned parallel workflow, partition safety rules, and batch manifest format. --- ## Command: `wiki absorb [date-range]` The core compilation step. Date ranges: `last 30 days`, `2026-03`, `2026-03-22`, `2024`, `all`. Default, with no argument, is `last 30 days`. If `raw/entries/` doesn't exist, run ingest first. Process entries one at a time. Read `_index.md` before each entry, match against existing articles, re-read every article before updating it, and identify what the new entry changes in your understanding. Write integrated sections or rich paragraphs, not bottom-appended scraps. Key rules: - **Anti-Dump (C4)**: synthesize raw material into wiki prose. Never paste raw entry text verbatim, never paste more than 3 consecutive raw lines, and keep the raw-to-prose compression ratio at least 5:1. - **Article creation**: named things with enough material get pages; patterns and themes get pages; do not create stubs. - **Anti-Cramming**: if you are adding a third paragraph about a sub-topic, that sub-topic probably deserves its own page. - **Anti-Thinning**: every touched page should get richer. - **Checkpoints**: rebuild index and backlinks, audit new article creation, review quality, split bloated pages, and adjust directories at the cadence required by the chosen scale mode. See `references/command-details.md` for the agent prompt template, full absorption loop, Anti-Dump examples and hard limits, article-creation rules, checkpoint cadence, and audit checklist. --- ## Command: `wiki query ` Answer questions about the subject's life by navigating the wiki. Read `_index.md`, use `_backlinks.json` to identify central topics, read only the relevant articles, and synthesize an answer from the compiled wiki. Query is read-only. Key rules: - Read the wiki, not `raw/entries/` - Lead with the answer and acknowledge gaps - Follow relevant wikilinks and `related:` entries surgically, not exhaustively - Do not modify any wiki files See `references/command-details.md` for the full answer procedure, query patterns by article type, and query rules. --- ## Command: `wiki remediate` Scan for raw entries that are not yet cited in any wiki article, and launch targeted remediation agents to close the coverage gap. Run this after `wiki absorb` and before `wiki cleanup`. Key rules: - Treat **citation coverage** and **content coverage** as separate metrics - Citation coverage is the completion gate and must reach 100% - Use the bundled coverage helpers, classify gaps, and remediate per batch - Add synthesized content when needed, but never violate Anti-Dump - Create supplementary articles only as a last resort See `references/command-details.md` for citation-versus-content coverage, process phases, when to run, and why remediation is separate from cleanup. --- ## Command: `wiki cleanup` Audit and enrich every article in the wiki using parallel subagents. Cleanup is broad restructuring work after coverage is stabilized. Key rules: - Build full wiki context from `_index.md` and every article - Assess structure, line count, tone, quote density, narrative coherence, wikilinks, and citations - Apply the orphan policy carefully. Orphans are allowed, but still try to connect them without over-linking - Restructure diary-driven pages into theme-driven narratives - After cleanup, deduplicate candidates, create needed pages, fix broken links, then rebuild index and backlinks See `references/command-details.md` for the cleanup agent prompt template, the three cleanup phases, orphan policy, and the Steve Jobs restructuring test. --- ## Command: `wiki breakdown` Find and create missing articles. Expands the wiki by identifying concrete entities and themes that deserve their own pages. Key rules: - Survey the wiki for bare directories, bloated pages, high-reference backlink targets without articles, and misclassified pages - Mine concrete entities and significant named things, not generic technologies or passing mentions - Rank candidates by reference count and classify them into directories - Create articles in parallel, then add backlinks from existing pages - With `--reorganize`, move misclassified articles to more appropriate directories See `references/command-details.md` for the breakdown agent prompt template, all four phases, and reclassification rules. --- ## Directory Taxonomy Directories emerge from the data. Don't pre-create them. The reference taxonomy covers 39 common directories across 7 groups: core, media and culture, inner life and patterns, narrative structure, relationships and people, work and strategy, and other. See `references/taxonomy.md` for the full directory list, types, and placement guidance. --- ## Writing Standards Key rules: - **The Golden Rule**: This is not Wikipedia about the thing. This is about the thing's role in the subject's life. - **C1 Linking**: use `[[filename_stem|Human Readable Title]]`. Obsidian resolves by filename, not by `title:`. - **C2 Filenames**: ASCII snake_case only. Keep display names in `title:` and `aliases:`. - **C3 Citation**: dual traceability. `sources:` is canonical and machine-readable. `## References` is the human-readable restatement. - **C4 Anti-Dump**: never paste more than 3 consecutive raw lines. Never exceed 150 content lines. Keep raw-to-prose compression at least 5:1. - **Tone**: Wikipedia. Flat, factual, encyclopedic. No peacock words, editorial voice, rhetorical questions, progressive narrative, or emotional adjectives. Direct quotes carry the feeling. - **Aliases**: include the filename stem, plain title, alternative names, and shortened forms. - **Schema**: required fields are `title`, `type`, `created`, `last_updated`, `aliases`, and `sources`. - **Narrative**: every article must have a point. Structure by role, meaning, phase, thesis, cycle, drift, reasoning, or setting rather than by diary chronology. See `references/writing-standards.md` for the full tone rules, frontmatter schema, article format, linking resolution test, filename convention, citation discipline, aliases discipline, narrative coherence, structure-by-type table, quote discipline, and length targets. --- ## Command: `wiki sync` Pull documentation from multiple source projects and update `raw/entries/`. This is the **collection layer** — it gathers source documents but does NOT run LLM absorption. Think of it as "wiki ingest, but for many sources at once." ### Quick Start 1. Create `sources.yaml` in the project root (see `references/sources-schema.md`). 2. Run: ```bash python scripts/sync_sources.py --config sources.yaml --wiki-root wiki/ ``` 3. Review new entries in `raw/entries/{source_name}/`. 4. Run `wiki absorb` (manually) to integrate new entries into wiki articles. ### What It Does 1. Reads `sources.yaml` for the list of source projects. 2. For each source: - `type: git` → sparse-clone to `.sync_cache/`, checkout only `doc_path` - `type: local` → read from local path directly - `type: obsidian` → delegate to `ingest_obsidian.py` (existing behavior) 3. Runs the appropriate ingest helper for each source: - `ingest_obsidian.py` for Obsidian vaults - `ingest_projects.py` for project `doc/` folders 4. Writes entries to `raw/entries/{source_name}/`. 5. Merges all per-source ingest logs into unified `raw/ingest_log.json`. 6. Runs `rebuild_index.py` and `check_coverage.py` automatically. 7. Reports sync summary with added/updated/unchanged/deleted counts. ### Incremental Sync After the first run, `sync_log.json` tracks each file's content hash. Subsequent runs only process changed files, skipping unchanged ones. - Use `--force` to ignore the cache and re-process everything. - Use `--dry-run` to preview changes without writing files. - Use `--source ` to sync only one specific source. ### Automation Levels This command handles **Level 1 (Source → Entries)** automation: | Level | What | Cost | Trigger | |---|---|---|---| | **Level 1** | Source docs → `raw/entries/` + index + backlinks | Seconds to minutes, no LLM | `wiki sync` (cron/timer/manual) | | **Level 2** | New entries → wiki article text | LLM cost per entry | `wiki absorb` (manual) | Level 2 requires manual `wiki absorb` — see Phase 3 in the implementation plan. ### ID Strategy Project sources use **source-prefixed IDs** to prevent collisions: ``` sha1(f"{source_name}:{relative_path}")[:12] ``` Obsidian sources keep the original ID strategy (`sha1(rel_path)[:12]`) to preserve backward compatibility with existing article citations. ### Deletion Handling When a source file is deleted: - The corresponding entry in `raw/entries/{source}/` is removed. - The entry is removed from `sync_log.json` and `ingest_log.json`. - Wiki articles that cited the deleted entry are NOT automatically modified. - Run `check_coverage.py` to identify orphaned citations. ### When to Run - After adding new docs to any source project's `doc/` folder - On a schedule (daily cron, systemd timer, GitHub Actions) - Before `wiki absorb` to ensure entries are up to date - After changing `sources.yaml` (adding/removing sources) ### See Also - `references/sources-schema.md` — Full `sources.yaml` schema documentation - `references/automation-guide.md` — GitHub Actions and systemd timer setup --- ## Command: `wiki rebuild-index` Rebuild `_index.md` and `_backlinks.json` from current wiki state. Exclude meta files, `README.md`, and optional `.wikiignore` entries. Generate index entries with `[[filename_stem|Display Title]]`, preserve aliases in the index line, and verify that every wikilink resolves through a filename or alias. See `references/command-details.md` for exclusions, required format, reference implementation, and verification steps. --- ## Command: `wiki reorganize` Step back and rethink wiki structure. Read the index, sample articles, and ask: merge, split, new categories, orphan articles, missing patterns. Execute changes, then rebuild index. See `references/command-details.md` for the full reorganize command note. --- ## Command: `wiki status` Report current wiki state using the standard structure: Ingestion, Articles, Coverage, Quality, and See also. See `references/command-details.md` for the exact output template. --- ## Principles 1. **You are a writer.** Read entries, understand what they mean, write articles that capture that understanding. 2. **Every entry ends up somewhere.** Woven into the fabric of understanding, not mechanically filed. 3. **Articles are knowledge, not diary entries.** Synthesize, don't summarize. 4. **Concept articles are essential.** Patterns, themes, arcs. These are where the wiki becomes a map of a mind. 5. **Revise your work.** Re-read articles. Rewrite the ones that read like event logs. 6. **Breadth and depth.** Create pages aggressively, but every page must gain real substance. 40 stubs is as bad as 5 bloated articles. 7. **The structure is alive.** Merge, split, rename, restructure freely. 8. **View photos.** Understand what they show and integrate them into the narrative. 9. **Connect, don't just record.** Find the web of meaning between entities. 10. **Cite sources.** Every claim traces back to a raw entry ID. --- ## Concurrency Rules - Never delete or overwrite a file without reading it first. - Re-read any article immediately before editing it. - Never modify `_absorb_log.json` directly. - Rebuild `_index.md` and `_backlinks.json` only at the very end of a command. - In partitioned mode, each agent owns exactly one directory and writes only there. - Cross-directory reads are allowed. Cross-directory writes are not. --- ## Migration from v1.0.0 See `references/migration.md` for backward-compatibility policy, recommended migration steps, when full migration is required, and lint warning versus error behavior.