How to Build a Simple RAG Pipeline in Python

RAG (Retrieval-Augmented Generation) is a fancy name for a practical idea: don't make the LLM guess if you can hand it the right context. Think of it like having a cheat sheet during an exam instead of cramming everything into your brain the night before.

Also: every startup I talk to asks about RAG right now, but most of them are overcomplicating it. I've built this same basic pipeline for multiple clients recently, and the winning move is usually "start simple, then get fancy later."

Four steps

1. Ingest documents (PDFs, markdown files, those random text files you've been hoarding).
2. Chunk them into smaller pieces.
3. Embed + index those chunks for similarity search.
4. Retrieve + generate an answer with citations.

The whole thing in one file

Below is a small, readable example using OpenAI embeddings + chat completions and a local FAISS index. FAISS is Facebook's gift to developers who don't want to pay for Pinecone.

You can swap out OpenAI for Anthropic or Gemini or Llama or another local model later. The structure stays the same.

# pip install openai faiss-cpu numpy
import os
import numpy as np
import faiss
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

docs = [
  ("pricing.md", "Our pricing is usage-based. The free tier includes 1k requests/day..."),
  ("security.md", "We encrypt data at rest using AES-256 and in transit using TLS 1.2+..."),
]

def chunk(text, size=200):
    words = text.split()
    for i in range(0, len(words), size):
        yield " ".join(words[i:i+size])

chunks, meta = [], []
for name, text in docs:
    for c in chunk(text):
        chunks.append(c)
        meta.append({"source": name, "text": c})

emb = client.embeddings.create(
    model="text-embedding-3-small",
    input=chunks
).data
X = np.array([e.embedding for e in emb], dtype="float32")

index = faiss.IndexFlatIP(X.shape[1])
faiss.normalize_L2(X)
index.add(X)

def retrieve(query, k=3):
    q = client.embeddings.create(
        model="text-embedding-3-small",
        input=[query]
    ).data[0].embedding

    q = np.array([q], dtype="float32")
    faiss.normalize_L2(q)

    scores, ids = index.search(q, k)
    return [meta[i] for i in ids[0]]

def answer(query):
    ctx = retrieve(query)

    context_text = "\n\n".join(
        [f"[{c['source']}]\n{c['text']}" for c in ctx]
    )

    prompt = (
        "Answer using ONLY the context below. "
        "If the answer is not in the context, say you don't know.\n\n"
        f"Context:\n{context_text}\n\n"
        f"Question: {query}"
    )

    resp = client.chat.completions.create(
        model="gpt-5.2-2025-12-11",
        messages=[{"role": "user", "content": prompt}],
    )

    return resp.choices[0].message.content, ctx

print(answer("How do you handle encryption?")[0])

What I wish I knew before my first RAG project

• Chunking matters. Too big and retrieval gets fuzzy. Too small and you lose context. Start with 150 to 300 words and refine as needed.
• Add metadata. Filenames, URLs, section headers. Future-you will want citations.
• Tell the model to only use the context. It'll still hallucinate sometimes. But it helps.
• If answers are bad, it's usually retrieval. People love blaming the model, but the real bug is often embeddings, chunking, or indexing.

If you want to go deeper, check out the FAISS docs and OpenAI's embedding guide. That's it.

Start simple. Add complexity when you actually need it. Most RAG pipelines I see are overengineered from day one, and then everyone wonders why it's slow, expensive, and "mysteriously" wrong.

If you're interested in practical AI applications or want to explore specialized systems working together, those approaches often work better than chasing AGI.

-Sethers