RAG That Cites Its Sources: Retrieval You Can Actually Check

Ask an internal assistant what the notice period is on the enterprise contract and it answers in two confident paragraphs. The number is wrong. Someone opens the source it names, a forty page master services agreement, and cannot find that number in it. The assistant did not cite a passage. It cited a filename.

That gap, between a citation you can open and one you have to trust, is the difference between a retrieval system and a demo.

What you index is what you can ever be right about

Ingest looks like plumbing and is actually policy. Every document you load becomes something the system asserts with equal confidence. Index the current, owned and canonical: the live handbook, the approved pricing sheet, the product docs, the signed contracts. Keep out drafts, superseded versions, the folder called "old", and transcripts where somebody speculated. Once a paragraph is a chunk, nothing tells the model it was a guess.

Permissions belong at index time, on the chunk, not at query time. Carry the source system's access list onto every chunk and filter inside the retrieval query, so a chunk a user may not read is never a candidate. The leak happens when a connector is pointed at a whole shared drive with a privileged service account: the index now holds the compensation spreadsheet and the restructuring plan, and an innocent question surfaces them. Run this check in ten minutes. Log in with the narrowest role you have and ask three questions only leadership should be able to answer.

Fixed-size chunking is where tables go to die

The default is to split every eight hundred tokens with a hundred of overlap. It is fast, and it cuts documents in the worst places. A pricing table splits across a boundary: the header row lands in one chunk and the numbers in the next. Retrieval returns the numbers, the model has no column names, and invents plausible ones. Headings fail the same way. "Notice period" sits at the bottom of one chunk and the clause it introduces at the top of the next, so the passage that answers the question no longer contains its words.

Structure-aware chunking splits on the document's own seams: headings, sections, list items, tables kept whole. Serialise a table as rows with the header repeated on each, so a row means something alone. Attach the heading path to every chunk and prepend it before embedding, so the chunk reads "MSA / Term and Termination / Notice" before its content.

Embedding is a commitment, and re-embedding is the bill

Every chunk becomes a vector under one specific model, and vectors from different models are not comparable. Changing the model means re-embedding the whole corpus, rebuilding the index, and either running two indexes or accepting a window of degraded answers. On a few million chunks that is real money and hours of pipeline time, and the bill arrives when a better model ships.

One decision makes that migration a weekend instead of a project: keep the parsed text and the chunk boundaries in your own storage, separate from the vector store. Re-embedding is then a re-run over text you already hold, not a re-ingest from a dozen connectors that have drifted since.

Pure vector search cannot find a part number

Dense retrieval matches meaning, which is what you want when "how do I cancel" finds a section headed "Termination". It is unreliable on exact tokens: a SKU, an error code, a contract number, a surname. Embeddings are lossy, and two identifiers differing by one character sit almost on top of each other, so the system returns the wrong part with total composure.

Hybrid retrieval is the usual answer. Run a keyword index, normally BM25, beside the vector index, query both, and fuse the ranked lists with reciprocal rank fusion, which needs no score calibration. Add metadata filters for what is fact rather than semantics: document type, product line, effective date. A ten minute check: take twenty real questions from your support inbox and count how many contain an identifier a human would copy and paste. In most B2B corpora it is a third to a half of them.

Reranking is the cheapest quality in the pipeline

First-stage retrieval optimises for recall, so let it. Pull fifty candidates cheaply, rerank them with a cross-encoder that reads the question and the passage together instead of comparing two vectors made in isolation, and pass the top five to eight into the prompt. The temptation is to skip that and stuff forty chunks into a long context window. Irrelevant chunks do not sit politely unused: they pull the answer toward whatever they say.

A citation has to point at a span, not at a filename

A citation is useful only if a reader can check it faster than they could have found the answer themselves. Every sentence carrying a fact carries a reference that resolves to a chunk identifier, then to a document, a section and a character range the interface opens with the passage highlighted. Those offsets must be recorded at index time; you cannot reconstruct them later. The pipeline also needs an abstention path: when nothing clears the relevance floor, the honest output is that the corpus does not contain the answer. A system that always answers sometimes fabricates.

"It hallucinated" is nearly always a retrieval failure

When an answer is wrong, look at the retrieved chunks first. Three cases, three fixes. The right passage was retrieved and the model misread it: a generation problem, addressed with the prompt or a stronger model tier. The right passage exists but was not retrieved: chunking, query construction, missing keyword search, or a reranker doing its job badly. The passage does not exist: a corpus problem, and no model solves it.

In production the second and third dominate. Teams that skip the triage go straight to changing the model, the most expensive intervention and the least likely cause. Measure retrieval separately from generation: hit rate at five, the share of questions where the correct passage appears in the top five, tells you which half to work on. Keep that in a fixed evaluation set, and keep the prompt that assembles the context in a registry with versions.

What retrieval cannot do for you

RAG cannot answer what nobody wrote down. The pricing exception in a sales director's head, the reason a clause exists, the process quietly abandoned in March: none of it is in the index, and the system answers from the stale written version without hesitating.

It degrades when the corpus contradicts itself. Two live policies with different numbers means retrieval returns one or both, and the answer is a coin flip delivered with confidence. Fix that by naming an owner and marking one document canonical, not by prompting.

And the quiet one: nobody maintains the corpus. Six months after launch the index still holds last year's pricing because no job flags what has not been reviewed. Give every source an owner and a review interval, and treat that as a running cost of the AI engineering layer, not a launch task.

Frequently asked questions

Why does a RAG system hallucinate even though it has the documents?

Usually because it did not receive the right documents. Retrieval runs before generation, and if the passage containing the answer never reached the top results, the model completes the pattern from its training. Inspect the retrieved chunks for a failed answer: if the correct passage is absent, the fault is chunking, keyword coverage or reranking. Only when the right passage was present and still misused is it a generation problem.

What is hybrid search and why does RAG need it?

Hybrid search runs a keyword index such as BM25 alongside the vector index and fuses the two result lists. Vector search matches meaning and is weak on exact strings, so part numbers, error codes and names get missed or confused with near-identical ones. Fusing both gives semantic recall and exact-match precision in one query, which is why hybrid is the default for business corpora.

How should chunk-level permissions work in a RAG pipeline?

The access list from the source system is copied onto each chunk when it is indexed, and the retrieval query filters on it before anything is scored. Filtering after retrieval, or relying on the model to refuse, is not a control. The common failure is pointing a connector at a whole shared drive with a privileged service account, which puts confidential files into an index everyone queries.

What should a citation in an AI answer link to?

A specific span in a specific document, not a document name. Record the chunk identifier and its character offsets at index time, attach a reference to every sentence carrying a fact, and have the interface open the source at the highlighted passage. A citation the reader has to search through makes checking slower than answering by hand.