Skip to content
PTEN

What is RAG? Retrieval augmented generation explained

What RAG (retrieval augmented generation) is, how the pipeline works, and when to pick RAG over fine-tuning, with examples from real projects.

RAG (retrieval augmented generation) is a technique where the system first searches your documents for relevant passages and then hands those passages to a language model along with the question. The model answers from what was retrieved, not just from what it learned in training. In practice, RAG is the most common way to make an LLM answer questions about data it has never seen: manuals, lessons, contracts, tax documents.

I've built RAG into a few projects: one over US tax documents, one over video lessons, and a customer service agent on WhatsApp. This post walks through what RAG is, step by step, and where each step tends to break.

What is RAG, in one technical sentence

The term comes from Lewis et al., 2020, which paired a document retriever with a text generator. Today the word covers any architecture with the same shape:

  1. Index: split documents into chunks, compute a vector (embedding) for each one, and store it.
  2. Retrieve: turn the question into a vector and find the closest chunks.
  3. Generate: build a prompt with the question and the chunks, and ask the model for an answer that cites its sources.

The model doesn't "learn" anything new. On every question it reads whatever context the search brought back. That's why a RAG system's quality depends more on retrieval than on the model.

How a RAG pipeline works

Ingestion and chunking

Before you can search, you need text. That sounds trivial and almost never is.

In IRS Helper, a RAG system over IRS forms and publications, crawling is done with Scrapy: separate spiders for PDFs and EPUBs from irs.gov. EPUBs are converted to PDF, and vectorization skips non-English versions of the same document so the index doesn't fill up with duplicates.

In the video lesson RAG, the text doesn't even exist at first. Uploads go into a BullMQ queue, and a worker in its own container extracts the audio with ffmpeg, transcribes it with Deepgram, and creates the embeddings. PDFs follow the same path.

Then comes chunking: cutting the text into pieces. In the video project, chunks are about 500 tokens with a 50-token overlap, and each one stores the start time of that passage in the lesson. That metadata is what later lets the answer cite the exact minute. The lesson I take from it: decide what a citation looks like before you decide how to chunk, because anything you don't store with the chunk won't come back later.

Embeddings and the vector store

An embedding is a vector of numbers that represents the meaning of a piece of text. Similar texts land close together in vector space. In my projects using OpenAI embeddings, the vectors have 1536 dimensions.

For storage and search I use PostgreSQL with pgvector. The upside is one less database: the chunk, its metadata, permissions and the vector live in the same table, and you can filter with plain SQL. pgvector offers two approximate indexes, and I've used both: HNSW in the video project and IVFFlat in DockPlus Atendimento.

Before that, at PowerX, I built RAG over internal documents using LangChain and Ollama, with the nomic-embed-text embedding model and pgvector. Same vector store; the embedding model is a swappable part.

A vector search in pgvector is just a SQL query. Generic example:

SELECT id, content, start_seconds
FROM chunks
WHERE course_id = $1
ORDER BY embedding <=> $2   -- cosine distance
LIMIT 20;

Retrieval: vectors, full-text and HyDE

Vector-only search misses exact terms. A vector captures "how do I report rental income" well, but it can miss "Form 8829", because the number is a rare token without much semantic meaning.

IRS Helper handles this before searching. If the question asks for a form ("Form 1040"), a regex pulls the form number and returns a direct link, with no LLM call at all. Only everything else goes to vector search.

In the video project, retrieval is hybrid: a vector search and a Postgres full-text search (tsvector) run over the same chunks, and the results are merged with Reciprocal Rank Fusion (RRF). RRF combines rank positions, not scores, so it doesn't matter that the two searches measure relevance on different scales.

The same project uses HyDE (Gao et al., 2022). The idea: a short question and a lesson transcript don't look much alike. So the model first writes a hypothetical answer, and the search uses that answer's embedding, which sits closer to the text you're trying to find. The hypothetical answer can be wrong; it's only bait for the search.

Reranking

Vector search is fast because it compares vectors computed ahead of time. A reranker is slower and more precise: it reads the question and each candidate together and scores how relevant they are.

The usual pattern is to retrieve many and rerank down to a few. In the video project, Cohere reranks the top 20 down to 5. In IRS Helper, also with Cohere, 10 go in and 5 come out. Five good passages fit in the prompt and leave less room for the model to get distracted.

Generation with citations

In the last step, the passages go into the prompt with an instruction: answer only from this context, and cite the source.

In the video project, citations come out as [Title, MM:SS], and the UI turns each one into a button that seeks the player to that point in the lesson. The student can check the answer in seconds. A good citation isn't decoration: it's what lets the user distrust the model.

The same system handles missing context. When the best retrieval score is below 0.65, the answer comes with a low-confidence warning. And each stage (HyDE, search, rerank, transcription) is logged as a span in Langfuse, so you can see where a bad answer started.

In IRS Helper, the answer goes out through a FastAPI server compatible with /v1/chat/completions, including stream: true. LiteLLM registers the server as a model named irs-helper, and OpenWebUI uses it like any other model. The RAG pipeline hides behind an interface every OpenAI client already speaks.

RAG inside an agent

RAG doesn't have to be the whole system. In DockPlus Atendimento, an agent for WhatsApp, Telegram and email, a LangGraph graph classifies the intent of each message and routes it to the right node. Questions go to the RAG node; scheduling requests go to booking; the rest can go to feedback or to a human.

That avoids a common mistake: sending every message through retrieval. "Can we move it to tomorrow?" doesn't need any document. Routing before retrieving saves calls and cuts down on answers that cite an irrelevant passage just because it happened to be the closest one.

RAG vs fine-tuning vs long context

Use RAG when the answer depends on facts that change, that belong to you, or that need a source. Updating knowledge means reindexing a document, not retraining. And you can enforce per-user permissions right in the SQL query.

Use fine-tuning when the problem is form, not facts: a fixed output format, a tone, a specific classification. Fine-tuning teaches behavior. It's a poor way to teach facts, because it can't cite a source and it goes stale the next day.

Use long context when the corpus is small enough to fit in the prompt, like one contract or a short manual. No index, no chunking, nothing in retrieval to go wrong. The cost is paying for every token on every question.

In practice the three combine. A large corpus behind RAG can still use a generous context window to send more passages.

Where RAG systems fail

  • Dirty input. A PDF with a broken table, a transcript with errors, the same page in three languages. Retrieval returns garbage with confidence.
  • Chunks without context. A piece that says "the deadline is 30 days" without saying for what. Store the title and section with the chunk.
  • Lost exact terms. Form codes, statute numbers, product names. Add full-text search, or a shortcut like the IRS Helper regex.
  • Questions that don't need retrieval. Route by intent first, as DockPlus Atendimento does.
  • Not enough context to answer. Set a score threshold and tell the user when the knowledge base doesn't cover the question, instead of letting the model improvise.
  • No way to debug. Without a log of each stage, a wrong answer won't tell you whether retrieval, reranking or the prompt was to blame.

Summary

RAG is search plus generation. The model is the easy part to swap; ingestion, chunking and retrieval are where the work is. Start with pgvector and plain vector search, measure where it fails, and only then add hybrid search, HyDE or reranking.

The projects mentioned here are on my projects page, and my background is on the about page. If you need RAG over your company's documents, get in touch.

All articles