---
name: forge-rag
description: Retrieval-augmented generation discipline. Semantic chunking with hierarchy, hybrid retrieval (BM25 + dense + rerank), citation discipline, refusal paths, eval-driven iteration, freshness handling. Contains chunking helper, hybrid-retrieval pseudocode, citation verifier. Use when designing a RAG pipeline or auditing one that returns wrong or hallucinated answers.
license: MIT
---

# forge-rag

You are building a RAG pipeline that has to answer real questions against a real knowledge base. Default agent-written RAG is "chunk by 1000 tokens, embed, top-k=5, stuff into context, done." Then it hallucinates because retrieval was wrong, chunks lost context, the model could not cite, and there is no eval to catch any of it. This skill exists to fix the whole pipeline.

The mental model: **RAG is a search system with an LLM front-end.** The LLM is only as good as the retrieval. Most RAG quality problems are retrieval problems, not generation problems.

## Quick reference (the things you must never ship)

1. Fixed-character chunking (e.g. every 1000 chars) ignoring semantic units.
2. Top-k=5 dense-only retrieval as the only retrieval strategy.
3. No reranking step.
4. No citation on generated claims.
5. No "I do not know" path in the system prompt.
6. Stuffing all retrieved chunks into context regardless of count.
7. No eval set covering refusal cases.
8. Same embedding model never re-embedded after a version change.
9. Cited chunk IDs not verified to be in the retrieved set.
10. Real-time indexing per user request.

## Hard rules

### Chunking

**1. Chunk by semantic unit, not character count.** Paragraphs, sections, function bodies, slide pages. A 1000-character window mid-sentence is the most common source of "system has the info but cannot answer."

**2. Preserve hierarchical context with each chunk.** A chunk from "Chapter 4 > Section 2 > Subsection 3" carries that hierarchy as metadata or as a prefix in the content.

```ts
type Chunk = {
  id: string;
  content: string;           // the actual passage
  metadata: {
    document_id: string;
    document_title: string;
    section_path: string[];  // ["Chapter 4", "Section 2", "Subsection 3"]
    position: number;
    mime_type: string;
  };
};

function withHierarchyPrefix(chunk: Chunk): string {
  return `${chunk.metadata.document_title} > ${chunk.metadata.section_path.join(" > ")}\n\n${chunk.content}`;
}
```

**3. Overlap chunks by 10-15% of length.** Strict boundaries cut concepts in half. Overlap recovers partial-match retrieval.

**4. Two-tier chunking for long documents.** Short "summary" chunk per section for broad retrieval, full sections for detail. Retrieve summary first; expand if needed.

**5. Tables, code, lists do not chunk like prose.** Keep intact or chunk by row/item. A truncated table is worse than a missing one.

### Embeddings

**6. Pick an embedding model that matches your domain.** General-purpose (text-embedding-3) for general text. Code-specific (Voyage Code, JinaAI Code) for code. Multilingual for multilingual.

**7. Embed the hierarchy prefix with the chunk content.** Improves "broad question" recall significantly.

**8. Embed both chunks AND hypothetical questions (HyDE).** Generate a hypothetical answer to the query, embed that, search against doc embeddings. Often beats embedding raw query.

**9. Re-embed when the embedding model version changes.** Embeddings from different models are not comparable.

### Retrieval

**10. Hybrid retrieval beats vector-only.** BM25 (keyword) + dense (vector) combined via Reciprocal Rank Fusion catches exact matches AND semantic matches.

```ts
// reference: RRF fusion
function reciprocalRankFusion<T extends { id: string }>(
  rankings: T[][],
  k = 60,
): T[] {
  const scores = new Map<string, { item: T; score: number }>();
  for (const ranking of rankings) {
    ranking.forEach((item, rank) => {
      const existing = scores.get(item.id);
      const increment = 1 / (k + rank + 1);
      if (existing) {
        existing.score += increment;
      } else {
        scores.set(item.id, { item, score: increment });
      }
    });
  }
  return [...scores.values()].sort((a, b) => b.score - a.score).map((s) => s.item);
}

// usage
const bm25Results = await bm25.search(query, { limit: 50 });
const denseResults = await vectorDb.search(embedQuery(query), { limit: 50 });
const fused = reciprocalRankFusion([bm25Results, denseResults]);
```

**11. Top-k starts at 20, not 5.** Retrieve generously, rerank aggressively. Top-5 dense retrieval misses too often.

**12. Rerank with a cross-encoder.** Cohere Rerank, Voyage Rerank, BGE Rerank. A cross-encoder scores query+document together; far more accurate than embedding similarity.

```ts
const reranked = await cohere.rerank({
  model: "rerank-english-v3.0",
  query,
  documents: fused.slice(0, 20).map((c) => c.content),
  topN: 8,
});
const final = reranked.results.map((r) => fused[r.index]);
```

**13. After reranking, pass 3-8 chunks to the model.** Too few miss context; too many trigger lost-in-the-middle.

### Query rewriting

**14. Rewrite ambiguous queries before retrieval.** User query "what about refunds?" is too short. Expand via an LLM call: "what is the refund policy for my recent purchase?"

**15. Decompose multi-part questions.** "Can I return this AND change my address?" → two retrievals, merge results.

**16. History-aware rewriting in multi-turn.** User: "show me the table." Bot: "<shows>." User: "what about the second row?" Rewrite to "what about the second row of the [table topic]?"

### Citation

**17. Cite the source for every claim.** Inline citation linked to chunk metadata.

```ts
// reference: system prompt fragment
const SYSTEM_PROMPT = `
You answer using only the passages in <context>.
For every factual claim, cite the chunk ID in [brackets] inline.
If the passages do not contain the answer, say "I cannot find this in the knowledge base."

<context>
${chunks.map((c) => `[${c.id}] ${withHierarchyPrefix(c)}`).join("\n\n")}
</context>
`.trim();
```

**18. Verify citations are real.** After generation, check that each cited chunk ID exists in the retrieved set. If not, regenerate or refuse.

```ts
function verifyCitations(answer: string, retrievedIds: Set<string>): {
  ok: boolean;
  invalid: string[];
} {
  const cited = [...answer.matchAll(/\[([\w-]+)\]/g)].map((m) => m[1]!);
  const invalid = cited.filter((id) => !retrievedIds.has(id));
  return { ok: invalid.length === 0, invalid };
}
```

**19. Return the cited chunks to the user, not just the answer.** UI shows "answer + source passages." Lets the user verify; lets you debug.

### Refusal

**20. The model is allowed to say "I do not know."** Bake into system prompt: "If the retrieved passages do not contain the answer, say so. Do not invent."

**21. Detect "no good chunks retrieved" before generation.** If top-1 rerank score below threshold, return "I could not find this" without calling the LLM at all. Saves cost AND avoids hallucination.

```ts
const THRESHOLD = 0.5;
if (reranked.results[0]?.relevance_score < THRESHOLD) {
  return { answer: "I could not find this in the knowledge base.", citations: [] };
}
```

### Freshness

**22. Index updates run on a schedule, not on demand per user.** Real-time indexing is expensive and rarely needed.

**23. Soft-delete in the index.** Mark documents removed but keep retrievable for one cycle in case deletion was a mistake. Hard-purge after grace.

**24. Versioned documents need version-aware retrieval.** Filter by version at retrieval time.

### Evaluation

**25. A RAG eval set has 100+ questions across diverse types.** Factoid, multi-doc, refusal-expected, ambiguous.

**26. Measure retrieval and generation separately.** Recall@K and MRR for retrieval. Faithfulness, answer relevance, citation correctness for generation.

**27. Test refusal behavior.** Half a dozen questions in the eval set should have no good answer in the KB; system should refuse, not invent.

**28. Re-run eval on every prompt/pipeline change.** Treat as regression test.

### Observability

**29. Log every query, retrieved chunks, generated answer, user feedback.** Without this, drift is invisible.

**30. Track retrieval coverage.** How often do users ask things not in the KB? Surfaces content gaps for the knowledge team.

## Common AI-output patterns to reject

| Pattern | Why wrong | Fix |
| --- | --- | --- |
| Chunk every 1000 chars | Cuts sentences | Semantic chunking by paragraph/section |
| Top-5 dense only | Misses keyword matches | Hybrid (BM25 + dense + rerank) |
| No reranker | Embedding similarity ≠ relevance | Cross-encoder rerank |
| Stuff all retrieved chunks into context | Lost-in-the-middle | 3-8 after reranking |
| No citation in prompt | Hallucination invisible | "Cite [chunk_id] inline" |
| No refusal path | Model invents answers | "Say 'I do not know'" in prompt + threshold check |
| Same embedding model across years | Stale, worse than current | Re-embed on model upgrade |
| Live indexing per user | Expensive, brittle | Scheduled batch indexing |
| Real-time indexing across versions | Mixes new + old answers | Filter by version |
| No eval set | Every change is vibes | 100+ questions, including refusals |

## Worked example: minimal RAG pipeline

```ts
import { z } from "zod";

const RAG_PROMPT = (chunks: Chunk[]) => `
You answer using only the passages in <context>.
For every factual claim, cite the chunk ID in [brackets] inline.
If the passages do not contain the answer, respond exactly: "I cannot find this in the knowledge base."

<context>
${chunks.map((c) => `[${c.id}] ${c.metadata.document_title} > ${c.metadata.section_path.join(" > ")}\n${c.content}`).join("\n\n")}
</context>
`.trim();

async function answer(query: string): Promise<{ text: string; citations: Chunk[] }> {
  // 1. Optionally rewrite the query
  const rewritten = await rewriteQuery(query);

  // 2. Hybrid retrieve
  const [bm25, dense] = await Promise.all([
    bm25.search(rewritten, { limit: 50 }),
    vectorDb.search(await embed(rewritten), { limit: 50 }),
  ]);
  const fused = reciprocalRankFusion([bm25, dense]);

  // 3. Rerank
  const reranked = await rerank(rewritten, fused.slice(0, 20));
  const top = reranked.slice(0, 6);

  // 4. Refusal-on-low-confidence (saves cost and hallucination)
  if ((reranked[0]?.score ?? 0) < 0.5) {
    return { text: "I cannot find this in the knowledge base.", citations: [] };
  }

  // 5. Generate
  const response = await llm.complete({
    model: "claude-sonnet-4-6",
    temperature: 0,
    system: RAG_PROMPT(top),
    messages: [{ role: "user", content: query }],
  });

  // 6. Verify cited chunks exist
  const retrievedIds = new Set(top.map((c) => c.id));
  const citationCheck = verifyCitations(response.text, retrievedIds);
  if (!citationCheck.ok) {
    logger.warn({ invalid: citationCheck.invalid }, "rag.hallucinated_citation");
    return { text: "I could not find this in the knowledge base.", citations: [] };
  }

  return { text: response.text, citations: top };
}
```

## Workflow

When building or auditing a RAG pipeline:

1. **Start with the eval set, even before the pipeline.** 30 questions you want answered.
2. **Build the simplest possible pipeline first.** Chunk, embed, vector search, generate. Measure baseline.
3. **Iterate retrieval first: hybrid, rerank, query rewrite.** Each step moves the eval needle.
4. **Add generation guardrails: cite, refuse-on-low-confidence, verify citations.**
5. **Add observability: log, sample, surface feedback.**
6. **Only when retrieval is at ceiling, tune the generation prompt.**

## Verification

Manual checklist - RAG quality is not grep-checkable:

- [ ] Eval set exists with 50+ questions including refusal cases.
- [ ] Retrieval is hybrid (or measured against hybrid).
- [ ] Reranking in the pipeline.
- [ ] Every answer carries citations.
- [ ] System prompt allows "I do not know."
- [ ] Logs capture query, chunks, answer for sampling.

## When to skip this skill

- Pure vector search (no LLM) - that is a search engine, different discipline.
- LLM apps with no retrieval.
- Frameworks that fully abstract the pipeline.

## Related skills

- [`forge-prompt-engineering`](../forge-prompt-engineering/SKILL.md) - the RAG system prompt.
- [`forge-evals`](../forge-evals/SKILL.md) - the eval set and rubric.
- [`forge-citation`](../../research/forge-citation/SKILL.md) - citation discipline.
- [`forge-llm-streaming`](../forge-llm-streaming/SKILL.md) - streaming the answer back.