A language model's knowledge comes entirely from what it saw during training. That creates real limitations once you try to build something useful with it.

• Knowledge cutoff — the model has no information about anything that happened after its training data was collected.
• Private data — a model has never seen your company's internal documents, policies, or product data, because that data was never public.
• Changing information — prices, policies, and product details change constantly; a model's internal knowledge is frozen at training time.
• Hallucination risk — when a model doesn't actually know something, it can still generate a fluent, confident-sounding answer that is wrong.
• Domain-specific knowledge — highly specialized or niche information is often underrepresented in general training data.

RAG doesn't try to make the model smarter or retrain it. It changes what information the model has available at the moment it answers, by retrieving relevant content and giving it to the model as context.

Without RAG, a model can only work with what it already knows. RAG adds one extra step before the model ever generates anything:

That extra step is the whole idea: before the model generates anything, the system searches for information relevant to the question and hands it to the model as part of the prompt.

Four ideas do most of the work here, and they map directly onto that sentence:

• Query — the question or request you want answered.
• Retrieval — searching a knowledge source for information relevant to that query.
• Context — the retrieved information, assembled into something the model can read alongside the question.
• Generation — the model producing an answer using the question and the retrieved context together.

Hold onto that before looking at implementation details: a retrieval step happens first, and its output becomes part of what the model reads before it answers.

At the moment a user asks a question, a RAG system runs a query-time pipeline:

That retrieval step only works because the knowledge source was prepared in advance. Before any question is ever asked, documents go through an ingestion pipeline:

Ingestion happens once, and again whenever documents change. Retrieval happens every single time a user asks a question.

Here's how the pieces from the diagrams above relate to each other:

Documents

The raw source material — PDFs, wiki pages, tickets, policies — that contains the knowledge you want the system to use.

Chunking

Splitting documents into smaller pieces so they can be embedded and retrieved effectively, instead of retrieving an entire document at once.

Embeddings

Numerical vector representations of text that let the system compare meaning, not just exact words.

Vector Database

Storage optimized for finding the embeddings most similar to a query's embedding, quickly, at scale.

Retrieval

The step of actually searching the knowledge source and returning the most relevant chunks.

Reranking

An optional second pass that reorders retrieved results using a more precise (and more expensive) model.

Context

The retrieved chunks, assembled into the prompt the model will actually read.

LLM

The model that reads the question and the retrieved context, and generates the final answer.

A company has thousands of internal documents — policies, handbooks, and process guides. An employee asks an internal assistant:

"What is our parental leave policy?"

A RAG system handling this:

1. Receives the question.
2. Searches the internal knowledge base for relevant content.
3. Retrieves the sections of the employee handbook that mention parental leave.
4. Sends those retrieved sections to the LLM as context, along with the original question.
5. Generates an answer grounded in the retrieved policy text.

This is a minimal illustration of the pattern, not a production implementation:

def answer_question(question, retriever, llm):
    # 1. Retrieve documents relevant to the question
    documents = retriever.search(question, top_k=3)

    # 2. Build context from the retrieved documents
    context = "\n\n".join(doc.text for doc in documents)

    # 3. Send the question and context to the model together
    prompt = f"""Answer the question using only the context below.

Context:
{context}

Question: {question}
Answer:"""

    return llm.generate(prompt)

Real systems add a chunking strategy, an embedding model, a vector database, metadata filtering, and often a reranking step — but the core pattern stays the same: retrieve, then generate.

• Poor chunking

  Chunks that are too large dilute relevance; chunks that are too small lose context. Both hurt retrieval quality.

• Retrieving irrelevant documents

  If the retriever returns weakly related content, the model will generate an answer grounded in the wrong information.

• Retrieving too much context

  Stuffing the prompt with excessive retrieved text increases cost and can bury the passage that actually matters.

• Ignoring metadata

  Filtering by source, date, or permissions is often as important as semantic similarity for returning the right content.

• Assuming vector search is always enough

  Keyword and hybrid search often outperform pure vector search for exact terms, IDs, or rare vocabulary.

• Skipping evaluation

  Without measuring retrieval quality and answer accuracy, it's hard to know whether changes actually improve the system.

• Confusing retrieval problems with LLM problems

  A wrong answer is often caused by bad retrieval, not a weak model — the model can only work with what it was given.

• Failing to inspect retrieved context

  The fastest way to debug a bad RAG answer is to look at what was actually retrieved, not just the final output.