RAG document-retrieval pipeline — classical IR baselines + modern vector RAG

Context

RAG implementations get asserted, not measured. “It works for our use case” is the modal claim. The build was a deliberate counterexample: a complete RAG pipeline alongside classical IR baselines, all benchmarked against the same published corpus so the lift from each stage is quantifiable.

Problem

A modern RAG stack (bi-encoder embeddings → vector store → top-k retrieval → cross-encoder rerank → LLM generation) makes a lot of architectural decisions implicitly. Without baselines, “the rerank helps” and “the bi-encoder is good enough” are both unfalsifiable. With baselines, they’re measurable.

Approach

Four independently-runnable sub-pipelines: classical Boolean retrieval, classical TF-IDF Vector Space Model with cosine similarity, modern dense retrieval with bi-encoder embeddings, and dense retrieval with cross-encoder reranking. All evaluated on the published CISI corpus (1,460 documents, 112 queries with ground-truth relevance judgments) with precision/recall scoring. Each pipeline benchmarkable in isolation, swappable surgically.

Solution

• Document ingestion with semantic chunking (not fixed-token windows).
• Embedding generation via HuggingFace Sentence Transformers: deliberate bi-encoder for recall, cross-encoder for precision.
• Local ChromaDB vector store with metadata-aware retrieval filtering.
• LangChain orchestration chaining retriever, reranker, and generator into a reusable pipeline.
• Classical Boolean retrieval with AND/OR/NOT operators implemented as a separate sub-pipeline.
• TF-IDF Vector Space Model with cosine similarity ranking implemented as a separate sub-pipeline.
• CISI-benchmark evaluation harness computing precision and recall against ground-truth relevance judgments.
• Four-sub-project codebase layout: each retrieval approach in its own folder, with pinned dependencies and step-by-step setup.

Outcome

Above. The “RAG works because the eval says it does” framing is the methodology contribution. The codebase is paste-ready as a reference architecture for documentation / knowledge-base RAG projects where the retrieval quality has to be defensible, not assumed.