Mastering Agentic AI: Knowledge Retrieval (RAG)

In my previous post, we looked at Human-in-the-Loop — keeping a person in the decision path for the calls an agent shouldn’t make alone. That pattern protects you from bad actions. But there’s an earlier failure mode it can’t fix: an agent that’s confidently, fluently wrong because the model is reasoning from training data that’s stale, incomplete, or just hallucinated.

A language model only knows what was in its training set. Ask it about your internal docs, last week’s incident, or a contract it has never seen, and it will still answer — smoothly, plausibly, and sometimes completely fabricated.

That’s the problem Knowledge Retrieval (RAG) solves. It’s the pattern that turns a model with a fixed, frozen brain into an agent that can look things up before it speaks.

Pattern #14: Knowledge Retrieval (RAG)
#

The Problem
#

LLMs have two hard limits baked in at training time:

A knowledge cutoff. The model knows nothing that happened after it was trained, and nothing that was never public — your wiki, your tickets, your codebase.
No source of truth. When the model doesn’t know, it doesn’t stay silent. It interpolates. The output is grammatically perfect and factually invented, and there’s no citation to check it against.

You can fine-tune to push new facts into the weights, but that’s expensive, slow, and goes stale the moment the facts change. What you actually want is for the agent to fetch the relevant facts at query time and reason over them — the same way a competent human answers a hard question by pulling up the right document first.

Retrieval-Augmented Generation is exactly that: retrieve relevant context, then generate an answer grounded in it. The model stops being the source of facts and becomes the thing that reasons over facts you hand it.

The Solution
#

RAG has two moving parts: a retriever that finds relevant context for a query, and a generator (the LLM) that answers using only that context. Two ways to wire it up, depending on how much structure you need.

1. The minimal RAG chain (LCEL) — When retrieval is a single step, an LCEL chain expresses the whole pattern declaratively. The key move is injecting retrieved context into the prompt before the LLM ever sees the question:

rag_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "You are a knowledgeable assistant. Answer the user's question based ONLY on "
     "the provided context. If the context doesn't contain enough information, say so.\n\n"
     "Context:\n{context}"),
    ("user", "{question}")
])

chain = (
    RunnablePassthrough.assign(context=lambda x: search_documents(x["question"]))
    | rag_prompt
    | llm
    | StrOutputParser()
)

The whole pattern is in those two blocks. RunnablePassthrough.assign runs the retriever and attaches the result as context while passing the original question through untouched. The system prompt does the other half of the work: “answer based ONLY on the provided context… if the context doesn’t contain enough information, say so.” That instruction is what converts a chatty model into a grounded one — it’s permission to admit ignorance instead of inventing.

In the example, search_documents is a stub over an in-memory dict, which is deliberate — it keeps the retrieval contract obvious:

def search_documents(query: str) -> str:
    """Simulates document retrieval from a search index."""
    query_lower = query.lower()
    results = []
    for key, docs in SEARCH_DB.items():
        if key in query_lower:
            results.extend(docs)
    if not results:
        for docs in SEARCH_DB.values():
            results.extend(docs)
    return "\n".join(f"- {doc}" for doc in results[:5])

Swap that function for a GoogleSearchAPIWrapper, a SQL query, or a vector store, and the rest of the chain doesn’t change. Retrieval is an interface, not an implementation.

2. Real retrieval with a vector store — Keyword matching only finds documents that share your exact words. Production RAG uses semantic search: embed your documents into vectors, embed the query the same way, and retrieve by similarity. That means splitting documents into chunks, embedding them, and indexing them in a vector database:

loader = TextLoader(local_file)
documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)

client = weaviate.Client(embedded_options=EmbeddedOptions())
vectorstore = Weaviate.from_documents(
    client=client,
    documents=chunks,
    embedding=GoogleGenerativeAIEmbeddings(model="models/embedding-001"),
    by_text=False
)
retriever = vectorstore.as_retriever()

Two parameters here earn their keep. chunk_size=500 controls how much text lives in each retrievable unit — too big and you dilute relevance, too small and you fragment meaning. chunk_overlap=50 repeats a sliver of text across chunk boundaries so a sentence that straddles a split isn’t lost. Tuning these is most of the unglamorous work of making RAG actually good.

3. RAG as a graph (LangGraph) — Once retrieval and generation are explicit, modeling them as a two-node graph makes the data flow legible and gives you somewhere to hang the next steps — re-ranking, grading, query rewriting:

class RAGGraphState(TypedDict):
    question: str
    documents: List[Document]
    generation: str

def retrieve_documents(state: RAGGraphState, retriever) -> RAGGraphState:
    print(f"--- RETRIEVING for: {state['question']} ---")
    docs = retriever.invoke(state["question"])
    return {"documents": docs, "question": state["question"], "generation": ""}

def generate_response(state: RAGGraphState, llm) -> RAGGraphState:
    print("--- GENERATING ---")
    context = "\n\n".join([doc.page_content for doc in state["documents"]])
    rag_chain = prompt | llm | StrOutputParser()
    generation = rag_chain.invoke({"context": context, "question": state["question"]})
    return {"question": state["question"], "documents": state["documents"], "generation": generation}

workflow = StateGraph(RAGGraphState)
workflow.add_node("retrieve", lambda state: retrieve_documents(state, retriever))
workflow.add_node("generate", lambda state: generate_response(state, llm))
workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "generate")
workflow.add_edge("generate", END)
app = workflow.compile()

retrieve → generate → END. It looks almost too simple to bother with a graph — and for this linear case, it is. But the moment you want agentic RAG — grade the retrieved docs, loop back and rewrite the query if they’re irrelevant, decide whether to retrieve at all — those are just extra nodes and conditional edges on this exact skeleton. The graph is the structure you grow into.

Why This Matters
#

RAG is the pattern that makes LLMs usable on private, current, and verifiable data. It shows up everywhere real agents do work:

Internal knowledge assistants: answer over your wiki, docs, and tickets — data the model was never trained on.
Customer support: ground responses in the actual product manual and policy, with citations the agent can point to.
Research and analysis: pull the relevant passages from a large corpus, then reason over them instead of from memory.
Code assistants: retrieve the relevant files and symbols before answering questions about a repo.

The honest trade-offs:

Retrieval quality is the ceiling. RAG is only as good as what the retriever returns. Garbage chunks in, confident garbage out. Chunking strategy, embedding model, and top-k matter more than the LLM you pick.
It adds latency and cost. Every query now does an embedding lookup plus a larger prompt. Stuffing five retrieved chunks into context isn’t free in tokens.
It doesn’t eliminate hallucination — it constrains it. The model can still ignore the context or over-extrapolate from it. The “answer ONLY from context, else say you don’t know” instruction is load-bearing, and you still want to verify on high-stakes answers.
Infrastructure. A real deployment means a vector store to run, an embedding pipeline to keep in sync, and a reindex story for when documents change.

Rule of thumb: reach for RAG whenever the right answer depends on facts the model can’t be trusted to have memorized — anything private, anything recent, anything you need to cite. If the task is pure reasoning over what’s already in the prompt, you don’t need it.

The Bigger Picture
#

This is post #14 in my series documenting Antonio Gulli’s Agentic Design Patterns. All credit for the conceptual work goes to him — I’m focused on producing clean, runnable Python implementations that you can clone, modify, and ship.

RAG ties back to several patterns we’ve already covered. It’s the concrete mechanism behind Tool Use when the tool is “search the knowledge base,” and it’s the production answer to the Memory pattern’s recall problem — long-term memory is a retrieval problem wearing a different hat. Parallelization composes cleanly on top: fan out retrieval across several stores, then synthesize.

All the code from this post is in my repository: carlosprados/Agentic_Design_Patterns, specifically under 14_Knowledge_Retrieval_RAG/. Both the LCEL search-based example and the LangGraph + Weaviate vector-store workflow run with uv run, support Gemini and Ollama via the shared get_llm() abstraction, and are ready to fork.

What’s Next
#

In the next post we’ll tackle Inter-Agent Communication (A2A) — what happens when a single agent isn’t enough and you need multiple agents, possibly built on different frameworks and running in different processes, to talk to each other through a shared protocol. RAG gives one agent access to knowledge; A2A gives a whole team access to each other.

Stay tuned.

Pattern #14: Knowledge Retrieval (RAG)#

The Problem#

The Solution#

Why This Matters#

The Bigger Picture#

What’s Next#