Healthcare patient-intake chatbot with safe-handoff

Context

A multi-clinic provider whose intake desk was the throughput bottleneck, and whose previous chatbot trial had been pulled because it tried to give triage advice. They wanted intake automation, not medical advice.

Problem

The risk wasn’t the happy path. The risk was the model “being helpful” on a question it had no business answering: symptom severity, dosing, “is this an emergency”. One bad answer would close the experiment for good.

Approach

A guardrail-first design: enumerate the conversational moves the bot is allowed to make, and refuse everything outside them. Symptom capture (record what the patient says) is allowed; symptom interpretation is not. Safe-handoff is a first-class action: when in doubt, the bot says “let me get a clinician” and pages on-call.

Solution

• Claude with a tightly-scoped system prompt + structured tool calls for the allowed moves (capture, schedule, request insurance, escalate).
• A separate classifier in front of every user turn that ran a red-flag check (chest pain, suicidal ideation, severe bleeding, pregnancy red-flags, paediatric red-flags); it bypassed the model entirely and paged on-call when triggered.
• Server-side conversation log with explicit consent capture and FHIR-shaped notes for the EHR.
• HIPAA-aligned hosting with BAAs in place; no patient data in model providers’ retention windows.
• 200-conversation eval set built with the clinical team, including adversarial prompts designed to extract triage advice.

Outcome

Above. The eval harness is what convinced the compliance team to sign off; they could see the safe-handoff firing on every case it should have.