---
name: knowledge-pipeline
description: Operate the local-first Weblyfe knowledge pipeline for ingesting documents and workspace content, embedding locally, semantic search, and graph exploration.
---

# knowledge-pipeline Skill

## Purpose

Operate the local-first Weblyfe knowledge pipeline: ingest documents and Google Drive / Notion content, embed locally, search by meaning, and surface it as a 3D graph in Mission Control. Built 2026-06-03. Codifies the patterns so future sessions and fleet agents do not re-derive them.

## Where it lives

`~/clawd/projects/knowledge-pipeline/`. Store at `.data/knowledge.db` (SQLite, gitignored). Drive cache `.data/drive-cache/`, Notion cache `.data/notion-cache/`.

## Core principle (from the adversarial review)

Local-first is REAL: SQLite + local embeddings are canonical. Pinecone is a deferred optional sync target, never the critical path. No client data is embedded until per-tenant isolation is hard. `tenant` is fail-closed (NOT NULL, scoped search never returns another tenant or UNASSIGNED).

## Quick reference

| Task | Command |
|------|---------|
| Ingest a local folder | `python3 ingest.py <dir> --tenant weblyfe-internal` |
| Search | `python3 query.py "your question" --tenant weblyfe-internal -k 5` |
| Sync a Google Drive folder | `python3 sync_drive.py <driveFolderId>` then ingest `.data/drive-cache/<id>` |
| Export Notion (streaming) | `NOTION_API_KEY=... python3 -u sync_notion.py` then ingest `.data/notion-cache --tenant notion` |

## Stack

- Embeddings: local Ollama `bge-m3` (1024-dim). `ollama pull bge-m3`. Helper `kp/embed.py` hits `localhost:11434/api/embed`. Nothing leaves the machine.
- Store: SQLite + brute-force numpy cosine (`kp/store.py`). Corpus is small (tens of thousands of chunks) so this is fast and avoids fragile native installs on Python 3.14. sqlite-vec / Chroma is the upgrade path.
- Parse: PyMuPDF (fitz, preinstalled) for PDF, plain read for md/txt (`kp/parse.py`). `iter_files` skips node_modules/.git/.next/etc.
- Incremental: files are skipped by md5 on re-run.

## Gotchas

- Python 3.14 (`/opt/homebrew/bin/python3`) has a broken pyexpat for native builds; chromadb/sentence-transformers installs are fragile. Stick to ollama + SQLite + numpy.
- Notion `/v1/search` returns thousands of page objects for a large workspace. Stream (write per page) instead of collecting all first, or it appears to hang.
- Notion rate limit is ~3 req/s; keep a 0.34s pause per request.
- nohup buffers stdout; use `python3 -u` for live progress.

## Mission Control integration

The 3D graph lives at MC `Memory > Graph` (reagraph WebGL, already installed). `src/app/api/memory/graph/route.ts` surfaces the pipeline DB as a `knowledge` cluster (GROUP BY path, cap 350 nodes). The Graph tab was gated `!isLocal`; unlocked in local mode in `memory-browser-panel.tsx`. Override the DB path with env `KNOWLEDGE_DB_PATH`.

## Next phases (not built)

Per-tenant namespaces then client folders; FastAPI read-only for MC; audio (faster-whisper) + OCR ingest; entity/relation knowledge graph behind a flag; Pinecone remote sync; clustering/analysis; Obsidian export; voice "ask the KB".