RAG (Retrieval-Augmented Generation) Skill

Purpose

Design and implement RAG pipelines that ground LLM responses in relevant, up-to-date knowledge. This skill covers the full pipeline: document ingestion, chunking strategies, embedding model selection, vector storage, retrieval with filtering, reranking for precision, and prompt construction with retrieved context. RAG reduces hallucination by giving the model factual context to work with.

Key Concepts

RAG Pipeline Overview

INGESTION (offline):
  Documents -> Chunking -> Embedding -> Vector Store

RETRIEVAL (at query time):
  Query -> Embed Query -> Vector Search -> Rerank -> Top-K Context

GENERATION (at query time):
  System Prompt + Retrieved Context + User Query -> LLM -> Response

EVALUATION:
  Context Relevance | Answer Faithfulness | Answer Relevance

Chunking Strategies

FIXED SIZE:
  Split at N characters/tokens with M overlap
  Simple but may break mid-sentence
  Good for: Uniform, well-structured documents

RECURSIVE CHARACTER:
  Split on paragraphs, then sentences, then words
  Respects document structure
  Good for: General purpose, articles, documentation

SEMANTIC:
  Split when embedding similarity between sentences drops
  Each chunk is semantically coherent
  Good for: Long documents with topic changes

DOCUMENT-AWARE:
  Split on markdown headers, HTML tags, code blocks
  Preserves structural meaning
  Good for: Technical documentation, code, structured content

RULES OF THUMB:
  Chunk size: 500-1000 tokens (sweet spot for most models)
  Overlap: 50-200 tokens (prevents losing context at boundaries)
  Smaller chunks = more precise retrieval, less context per chunk
  Larger chunks = more context per chunk, less precise retrieval

Embedding Model Selection

MODEL                    DIMENSIONS  SPEED    QUALITY  COST
text-embedding-3-small   1536        Fast     Good     Low
text-embedding-3-large   3072        Medium   Best     Medium
voyage-3                 1024        Fast     Great    Medium
cohere-embed-v3          1024        Fast     Great    Medium
BGE-large (open source)  1024        Self-host Good    Free (compute)

GUIDELINES:
  - Start with text-embedding-3-small for prototyping
  - Upgrade to text-embedding-3-large or voyage-3 for production
  - Use the SAME model for document and query embeddings
  - Matryoshka embeddings: can truncate dimensions for speed

Patterns

Document Ingestion Pipeline

import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';
import { OpenAIEmbeddings } from '@langchain/openai';

// 1. Load and split documents
const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 800,
  chunkOverlap: 200,
  separators: ['\n## ', '\n### ', '\n\n', '\n', '. ', ' '],
});

const chunks = await splitter.splitDocuments(documents);

// 2. Embed chunks
const embeddings = new OpenAIEmbeddings({
  model: 'text-embedding-3-small',
  dimensions: 1536,
});

const vectors = await embeddings.embedDocuments(
  chunks.map((chunk) => chunk.pageContent)
);

// 3. Store in vector database
for (let i = 0; i < chunks.length; i++) {
  await vectorStore.upsert({
    id: `doc_${chunks[i].metadata.source}_chunk_${i}`,
    values: vectors[i],
    metadata: {
      text: chunks[i].pageContent,
      source: chunks[i].metadata.source,
      section: chunks[i].metadata.section,
      chunkIndex: i,
    },
  });
}

Retrieval with Hybrid Search

async function retrieve(query: string, options: {
  topK?: number;
  filter?: Record<string, unknown>;
  minScore?: number;
} = {}): Promise<RetrievedChunk[]> {
  const { topK = 10, filter, minScore = 0.7 } = options;

  // 1. Embed the query
  const queryVector = await embeddings.embedQuery(query);

  // 2. Vector search
  const vectorResults = await vectorStore.query({
    vector: queryVector,
    topK: topK * 2,  // Fetch more for reranking
    filter,
    includeMetadata: true,
  });

  // 3. Filter by minimum score
  const filtered = vectorResults.matches.filter(
    (match) => match.score >= minScore
  );

  // 4. Rerank for precision
  const reranked = await reranker.rerank(query, filtered.map(m => m.metadata.text));

  // 5. Return top-K after reranking
  return reranked.slice(0, topK).map((result) => ({
    text: result.text,
    score: result.relevanceScore,
    source: result.metadata.source,
  }));
}

RAG Prompt Construction

async function generateWithRAG(userQuery: string): Promise<string> {
  // 1. Retrieve relevant context
  const chunks = await retrieve(userQuery, { topK: 5 });

  // 2. Build context block
  const context = chunks
    .map((chunk, i) => `[Source ${i + 1}: ${chunk.source}]\n${chunk.text}`)
    .join('\n\n---\n\n');

  // 3. Generate response with context
  const response = await llm.create({
    model: 'claude-sonnet-4-20250514',
    system: `You are a helpful assistant. Answer the user's question based on the provided context.

Rules:
- Base your answer ONLY on the provided context
- If the context does not contain enough information, say so
- Cite sources using [Source N] notation
- Do not make up information not present in the context`,
    messages: [
      {
        role: 'user',
        content: `Context:\n${context}\n\n---\n\nQuestion: ${userQuery}`,
      },
    ],
  });

  return response;
}

PostgreSQL pgvector (Self-Hosted Vector Search)

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create embeddings table
CREATE TABLE document_chunks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  content TEXT NOT NULL,
  embedding vector(1536),
  metadata JSONB DEFAULT '{}',
  source TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Create HNSW index for fast similarity search
CREATE INDEX ON document_chunks
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

-- Similarity search query
SELECT id, content, source, metadata,
       1 - (embedding <=> $1::vector) AS similarity
FROM document_chunks
WHERE metadata->>'category' = $2  -- optional filter
ORDER BY embedding <=> $1::vector
LIMIT 10;

Evaluation

RAG Quality Metrics

CONTEXT RELEVANCE:
  Are the retrieved chunks relevant to the query?
  Measure: LLM-as-judge or human annotation

ANSWER FAITHFULNESS:
  Is the answer grounded in the retrieved context (no hallucination)?
  Measure: Check if every claim maps to a context chunk

ANSWER RELEVANCE:
  Does the answer actually address the user's question?
  Measure: LLM-as-judge on query-answer alignment

CONTEXT PRECISION:
  Are the top-ranked chunks the most relevant?
  Measure: Precision@K with ground truth labels

CONTEXT RECALL:
  Did retrieval find all relevant chunks?
  Measure: Recall against known relevant documents

Best Practices

1. Chunk size matters -- 500-1000 tokens is the sweet spot; too small loses context, too large dilutes relevance
2. Always use overlap -- 10-20% overlap prevents losing information at chunk boundaries
3. Preserve metadata -- source, section headers, and document type enable filtering
4. Rerank after vector search -- vector similarity is approximate; reranking improves precision
5. Use hybrid search -- combine vector similarity with keyword matching for best results
6. Test with real queries -- build a test set of questions with known answers
7. Monitor retrieval quality -- log what gets retrieved; poor retrieval causes poor generation
8. Keep embeddings fresh -- re-embed when source documents change
9. Use the same embedding model -- query and document embeddings must use the same model
10. Cite sources -- always attribute retrieved information to its source

Common Pitfalls

Pitfall	Impact	Fix
Chunks too large	Retrieval returns irrelevant content	Reduce chunk size to 500-800 tokens
No overlap between chunks	Context lost at boundaries	Add 100-200 token overlap
Different embed models for docs vs queries	Terrible retrieval quality	Use the SAME model for both
No metadata filtering	Irrelevant results from wrong documents	Store and filter on source, category, date
Trusting vector search alone	Low precision for some queries	Add reranking step
No evaluation	No idea if RAG is working well	Build a test set, measure faithfulness
Stale embeddings	Answers based on outdated information	Re-embed on document updates