RAG Architecture: A Practical Explanation
Breaking down Retrieval-Augmented Generation — from document chunking and embedding to vector search and LLM context injection.
Retrieval-Augmented Generation (RAG) is the industry standard for feeding private knowledge to large language models. It provides context without the high cost of fine-tuning. But how does it work under the hood? Let's trace a document through the entire RAG pipeline.
The 3 Stages of RAG
1. Ingestion (Chunking & Embeddings)
A document is too large to fit in a prompt, so we must slice it into manageable pieces (chunks). A good chunk size is 500-1000 characters, with an overlap of 100 characters to keep context unbroken across boundaries. Each chunk is passed to an embedding model (like text-embedding-3-small) to get a numerical vector representation.
2. Indexing & Storage
The vectors are stored in a dedicated vector index (like ChromaDB, Pinecone, or pgvector). This organizes the data so it can be searched mathematically in N-dimensional space.
3. Retrieval & Generation
When a user asks a question, the question is converted into a vector using the same embedding model. We search the vector index for the top-K chunks with the highest cosine similarity (closest vectors). Finally, these chunks are combined into a system prompt for the LLM:
System: Use the following context to answer the user's question.
Context: [Retrieved Chunk 1] \n [Retrieved Chunk 2]
User Question: [User Prompt]
By implementing a local RAG stack in FRIDAY, we were able to provide private document-searching capabilities with zero leakage of proprietary data to public clouds.