Custom LLMs for a domain knowledge base

Context

A mid-sized SaaS company with a sprawling internal knowledge base (Confluence, Notion, Slack archives, dozens of PDFs) wanted a single answer interface for new hires and support engineers. Off-the-shelf “chat with your docs” tools had degraded into hallucinations once the corpus grew past ~5,000 documents.

Problem

The two failure modes the client wanted gone: (1) confidently-wrong answers that referenced documents that didn’t exist, and (2) “I don’t know” responses on questions whose answer was in the corpus but in a format the retriever couldn’t see (tables, screenshots, code).

Approach

The strategy that won: smaller fine-tune for tone + structure, large pretrained model for reasoning, RAG for grounding, with cite-or-refuse as the core rule. If the retriever didn’t surface a confident hit, the model was instructed to say so and link the closest match for a human to follow up. No graceful hallucinations.

Solution

• Document ingestion pipeline: chunked at semantic boundaries (not fixed token windows), embedded with two complementary models, indexed in Pinecone.
• A reranker pass on the top-50 retrieval to recover answers buried in tables and code blocks.
• Lightweight fine-tune on the company’s voice, primarily for first-person plural framing and avoiding marketing language.
• Citation enforcement: every answer carries a “Sources” section with direct links to the source documents and section headings.
• Eval harness with a curated 200-question set ran on every prompt or retriever change.

Outcome

Above (in frontmatter). The team kept the eval harness and now uses it to gate every prompt or model change. That’s the part the client found kept coming back to in retrospect.