RAG refactor: from 64% to 91% answer accuracy in three weeks

Context

A 35-person SaaS had shipped an internal “ask the docs” bot a year earlier, on GPT-3.5 with a basic vector retriever. The support team used it for a quarter, lost trust in it, and went back to grepping Notion. By the time the team called DatVerse, the bot was still in the product but nobody on the support side opened it.

Problem

The retriever was the bug, not the model. A 200-question benchmark the client put together showed 64% answer accuracy; most failures were either confidently-wrong answers where the retriever had missed, or refusals on questions whose answer was sitting in the corpus but in a format the embedder couldn’t see (tables, code blocks, threaded Slack screenshots that had been pasted as PDFs).

Approach

Three-week scope, no greenfield rewrite. The strategy was: keep the chat UI, keep the LLM, keep the database. Replace the retrieval layer with a hybrid BM25 + vector store, add a reranker pass, and bolt a cite-or-refuse rule onto the prompt. Then hand the support team an eval harness they could run themselves.

Solution

• Hybrid retrieval. Postgres + pgvector for embeddings, alongside a Postgres tsvector BM25 index. Both run in parallel, top-k from each, deduped by source.
• Reranker. A small cross-encoder run on the top-50 to recover answers buried in tables and code.
• Cite-or-refuse prompt. Every answer either includes a Sources: section with direct links to the source docs, or refuses and points at the closest near-miss for the support engineer to follow up.
• Eval harness. The 200-question benchmark the client built moved into a small FastAPI eval runner that fires on every prompt or retriever-config change in CI. Pass/fail diffs are posted to the team’s Slack.

Outcome

The accuracy number is in the headline metrics. The follow-on outcome the client cared more about: two weeks after handoff, the support team had reopened the bot in their daily flow without anyone asking them to. The eval suite is what made that switch credible; they could see exactly which questions had gotten better and which had not.