RAG with FAISS: chunking, MMR and re-ranking

Naive top-k retrieval returns near-duplicate chunks and misses the passage that answers the question. Chunk deliberately, choose a FAISS index that matches your scale, diversify results with Maximal Marginal Relevance, then re-rank with a cross-encoder before context reaches the model.

Why does naive top-k retrieval fail?

Rank a policy corpus by cosine similarity and take the top five, and you often get five near-duplicates: three restatements of one clause, one superseded revision of it, and nothing from the exception paragraph next to it. Pure similarity search has no concept of redundancy — it scores each chunk against the query alone, so a boilerplate paragraph repeated across forty documents crowds out the one chunk that would have answered the question differently.

That is worse than an annoyance in a regulated setting. An assistant that quotes the wrong revision of a policy is not just unhelpful — it is a wrong answer delivered with apparent confidence. Retrieval quality is the ceiling on generation quality: no prompt recovers a fact that never made it into context.

How should you chunk documents for retrieval?

Chunking decides what unit of text a similarity score is computed against, and it matters more than the embedding model. Four strategies cover most production pipelines.

Strategy	Typical size	Overlap	Best for
Fixed-size	~512 tokens	~10% (≈50 tokens)	Homogeneous plain text — FAQs, chat logs
Recursive (structure-aware)	~800 characters	~100–200 characters	Prose with headings and paragraphs — most policy and manual text
Semantic	Variable, by topic shift	None (breakpoint-cut)	Long documents where topics change mid-section
Parent-child	Child 200–400 tokens, parent 1500–2500 tokens	Small overlap on the child only	Precise matching with full context returned to the model

Fixed-size splitting is the cheapest and the crudest: cut every 512 tokens with a 10% overlap so a fact split across the boundary still appears whole in a neighbouring chunk. It works on flat, repetitive text, and badly on anything with structure, because it cuts a clause in half as often as it respects one.

Recursive splitting — LangChain's RecursiveCharacterTextSplitter is the common implementation — tries a list of separators in order, from double newline down to single character, and only falls back to a harder cut when a paragraph is too long to keep whole. Around 800 characters with 100–200 of overlap keeps most clauses intact.

Semantic chunking embeds consecutive sentences and cuts where similarity between neighbouring sentence embeddings drops, so a boundary lands where the topic actually changes rather than where a character counter runs out. It costs one extra embedding pass but earns its keep on long documents that drift across several subjects.

Parent-child chunking separates what you search from what you serve: index small child chunks (200–400 tokens) for retrieval precision, but store a pointer from each to a larger parent section (1500–2500 tokens). A query matches the precise child; the pipeline hands the model the parent, so the answer keeps context a 300-token fragment would not. Default to this once naive top-k has already caused one wrong answer in review.

Which FAISS index fits: Flat, IVF or HNSW?

Three index families cover almost every RAG corpus size. IndexFlatIP is exhaustive: it compares the query against every stored vector, so cost grows linearly, but the result is exact and reproducible — the same query returns the same ranking every time, which matters when an auditor asks why the assistant retrieved a given clause. It needs no training step and is the default until profiling says otherwise, usually up to roughly a million chunks.

IndexHNSWFlat is a graph: each vector links to a bounded number of neighbours, and search walks the graph instead of scanning every vector. It needs no training, and stays low-latency well past a million vectors, at the cost of more memory per vector than a compressed IVF index and no support for removing vectors — a corpus with hard deletions needs a rebuild, or IVF instead.

IndexIVFFlat and its compressed sibling IndexIVFPQ partition the vector space into clusters with k-means, then search only the nearest few (nprobe) instead of the whole set. They need a training pass first. IVF earns its complexity once the corpus is too large for HNSW's memory footprint, or once PQ compression is needed to fit the vectors at all — FAISS's own guidance is that IVF becomes the practical option once full-precision vectors no longer fit in memory.

import faiss
import numpy as np

d = 384  # e.g. sentence-transformers all-MiniLM-L6-v2
xb = np.load("chunk_embeddings.npy").astype("float32")
faiss.normalize_L2(xb)  # inner product now behaves like cosine similarity

# Exact, exhaustive -- reproducible, good for an audit trail
index_flat = faiss.IndexFlatIP(d)
index_flat.add(xb)

# Graph-based -- no training, low query latency, more RAM per vector
index_hnsw = faiss.IndexHNSWFlat(d, 32, faiss.METRIC_INNER_PRODUCT)
index_hnsw.hnsw.efSearch = 64
index_hnsw.add(xb)

# Clustered -- needs training, scales past what HNSW can hold in RAM
quantizer = faiss.IndexFlatIP(d)
index_ivf = faiss.IndexIVFFlat(quantizer, d, 100, faiss.METRIC_INNER_PRODUCT)
index_ivf.train(xb)
index_ivf.add(xb)
index_ivf.nprobe = 8  # cells searched per query -- trades recall for speed

How does Maximal Marginal Relevance add diversity?

Maximal Marginal Relevance, from Carbonell and Goldstein's 1998 paper on summarisation and passage re-ranking, re-orders a candidate set to reward relevance and penalise similarity to what is already picked:

MMR = argmaxDi ∈ R \ S [ λ · Sim1(Di, Q) − (1 − λ) · maxDj ∈ S Sim2(Di, Dj) ]

R is the candidate set FAISS returned, S is the set already selected, Q is the query, and λ trades relevance against diversity: λ = 1 is plain similarity ranking, λ = 0 picks purely for novelty. In practice, pull 50–100 candidates from FAISS, run MMR to narrow them to a diverse 20–30, and let the re-ranker choose the 4–8 chunks that go into the prompt. A λ around 0.5–0.7 keeps enterprise Q&A leaning towards relevance while still forcing out at least one non-duplicate chunk.

import numpy as np

def mmr(query_vec, candidate_vecs, k=5, lambda_mult=0.6):
    """Greedy MMR over L2-normalised embeddings (dot product = cosine sim)."""
    selected, remaining = [], list(range(len(candidate_vecs)))
    sim_to_query = candidate_vecs @ query_vec
    while remaining and len(selected) < k:
        if not selected:
            pick = remaining[int(np.argmax(sim_to_query[remaining]))]
        else:
            sim_to_selected = candidate_vecs[remaining] @ candidate_vecs[selected].T
            redundancy = sim_to_selected.max(axis=1)
            score = lambda_mult * sim_to_query[remaining] - (1 - lambda_mult) * redundancy
            pick = remaining[int(np.argmax(score))]
        selected.append(pick)
        remaining.remove(pick)
    return selected  # indices into candidate_vecs, in MMR order

When does cross-encoder re-ranking earn its latency?

The embedding model behind FAISS is a bi-encoder: it scores the query and each chunk independently, which is what makes indexing millions of chunks feasible. A cross-encoder feeds the query and one candidate chunk into the same transformer together, so it attends across both texts before producing a relevance score — far more accurate, and far slower, so it runs against the 20–30 candidates MMR narrowed down, never the whole corpus.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, chunk.text) for chunk in shortlisted_chunks]
scores = reranker.predict(pairs)

ranked = [c for _, c in sorted(zip(scores, shortlisted_chunks), key=lambda p: -p[0])]
context = ranked[:6]

Re-ranking the whole corpus with a cross-encoder defeats the reason you built a vector index in the first place.

How do you evaluate the pipeline, and what should you log?

Two numbers catch most regressions. Recall@k needs a labelled set of question and correct-chunk pairs you maintain yourself: the fraction of questions where the correct chunk lands in the top k, measured before and after re-ranking, so a drop after re-ranking shows a broken filter. Faithfulness checks whether each claim in the generated answer is actually supported by retrieved context rather than the model's prior knowledge — I cover it in more depth in the evaluation frameworks piece on this blog; frameworks such as Ragas compute it by extracting claims from the answer and checking each against context. Then log, per request:

• the query text, or a hash of it if the content is sensitive
• the chunk ids and raw similarity scores FAISS returned
• the MMR-selected subset and the λ used
• the cross-encoder scores, if re-ranking ran
• the final chunk ids sent to the model, and the model name and version
• the generated answer and the latency of each stage

That log, not the demo, is what an incident review or an audit will ask for. Build it before launch.

Can this run on-premises when data may not leave the bank?

FAISS itself has no network dependency — it is a CPU or GPU library that runs in-process, so once the index is built inside the perimeter, it and the search step never leave. The parts of a RAG pipeline that usually do reach outside are the embedding model and the generation model. An open-weights embedding model such as BAAI/bge or E5, served locally through sentence-transformers or a small inference server, replaces a hosted embeddings API. An open-weights chat model served the same way replaces the hosted LLM. The cross-encoder re-ranker is a similarly small model and stays local for the same reason — none of the document text or embeddings has to cross the perimeter to be scored.

What is left is not a library choice but a set of decisions: where the index is stored and backed up, who is authorised to call the retrieval service, and whether the query log counts as personal or client data under the bank's classification. Treat those the way an agent's tool calls are treated in review — an allow-list, a log line per request, least privilege — rather than something the library resolves on its own.

Practical takeaways

1. Chunk with structure in mind first; use fixed-size windows only as a fallback for plain, unstructured text.
2. Start with IndexFlatIP; move to HNSW or IVF once latency or memory forces it, and keep an exact index for anything that must be reproducible on audit.
3. Pull more candidates than you need, diversify with MMR, then re-rank the shortlist with a cross-encoder — never go straight from raw similarity to the prompt.
4. Track recall@k before and after re-ranking, and a faithfulness score on generated answers, against an evaluation set you maintain yourself.
5. Log every stage of the pipeline per request; that log is what will be reviewed, not the demo.

Sources

• FAISS wiki — Guidelines to choose an index
• FAISS wiki — Faiss indexes
• Carbonell, J. & Goldstein, J. (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. SIGIR '98.
• Sentence Transformers — Retrieve & Re-Rank
• Ragas — Available metrics (faithfulness, context recall)

Sources checked September 2026.