For a top-5 national health insurer's payment-integrity program. Fourteen AI components analyze every claim through an 8-stage pipeline; auditors work the output through a conversational review surface that remembers what they asked yesterday and captures every override as training data.
I'd been embedded in a top-5 insurer's cost-of-care program for nearly two years — chatbots first, then dashboards, then the audit work itself — meeting daily with the people who actually write and run audit edits. That's where the real problem surfaced. An audit verdict looks like one clean decision; underneath it is a data problem. Medical records are barely structured — about a tenth live in the EMR, the rest are PDFs, faxes, and feeds from outside systems — HIPAA-gated, and they never arrive complete or all at once, so cases stall waiting for records. And this is prepay: the call has to be right before the money goes out, not clawed back after — which is unforgiving, because a wrong call is a provider and a member you've harmed.
Same 14-component pipeline underneath; four very different surfaces on top. The auditor owns the decision; the manager owns the pattern; the executive owns the outcome; platform ops owns the machine.
Follow one auditor — John — from login to decision, then zoom out to the platform that ran underneath him. Every panel below is live React, reconstructed from the production app.
John doesn't open to an empty work list. While he was gone, the platform screened all 19 overnight claims and ranked them by SLA risk, dollar value, and pattern alerts — then opened with what to do first. The morning brief is AI-driven; the prioritization is the product.
I've screened 19 claims overnight — 3 SLA-critical, 6 high-priority, and 6 AI pattern alerts need your review.
John enters the case workspace for CLM-IMP-209024 — a $48,750 implant claim from Central Spine & Joint. Everything about the claim is here, but the core is two things: on the left, what the Audit Agent found (summary, core findings, AI findings, precedent); on the right, a Chatbot he can interrogate. Open any finding and ask it Policy check, Evidence, or Reasoning, or tap a similar case — the answers type out and accumulate as a conversation. (Reconstructed for this case study — not the production screens.)
When the auditor aligns with the AI, one click confirms and the denial reason pre-populates. When they override — approve a claim the AI said to deny, or escalate what it approved — a justification is required. That note flows to the learning loop as labeled training data. Try it:
Everything John just did ran on a multi-agent pipeline. The Agent Ops View is the surface for the technical team that runs it — the way a human manager watches team performance, this lets them watch the AI: the architecture, the workflow, and real-time status. Two lenses. By Architecture answers “is the machine running?” — for each stage, how many cases it's holding, average cycle time, whether a step is degraded or a bottleneck, and where cost concentrates. By Case Performance answers “are the decisions any good?” — pull one claim as a spot-check and replay exactly how the AI handled it, stage by stage: the execution logs, the chain of thought, and how it handled errors. This is the surface I designed closest with engineering — I had to understand each agent's reasoning and failure modes before I could make them legible.
Pipeline is healthy overall — 412 claims/hr, $2,840 compute spend, SLA 94.2%. Two items to watch: Quality Check error rate is 1.85% (above the 1% threshold — LLM-judge calibration drift), and Audit Agent cost is up 22% week-over-week on longer implant claims. Human Review is the bottleneck (64 queued). Everything else is within range.
The reasoning core — a 7-step LLM workflow. This stage's own cost is climbing on longer, document-heavy implant claims; token cost is the thing to watch.
Override justifications and experience scores feed the Learner Agent, which updates the Intelligence Library and per-audit-type ML configs.
When a stage drifts (e.g. Quality Check above), the config loop recommends and ships a new configuration — no redeploy.
A wrong call before payment harms a provider and a member, with no clawback to undo it. So every design choice here serves one goal: a decision trustworthy enough to act on before the money goes out. That accuracy pays back three ways — improper charges prevented at prepay, more claims worked per auditor hour, and fewer appeals because an aligned decision auto-drafts an accurate denial letter. And because every override loops back to the agent as labeled feedback, accuracy compounds — which is what makes prepay safe to expand.
Three levers: improper charges prevented at prepay instead of recovered months later; more claims worked per auditor hour; and fewer appeals, because an aligned decision auto-drafts an accurate denial rationale and letter. Together, the program's biggest impact lever this year.
The agent points straight to the problem; the auditor verifies in a glance instead of hunting across reports — and when trust is high, doesn't even open the attached PDF. Minutes per claim instead of tens of minutes.
When the AI recommends and the auditor decides, they align 9 out of 10 times. The 9% disagreement is the most valuable training signal the model gets.
Every disagreement with the AI requires a justification note. Not compliance theater — the note becomes labeled data for the learning loop.
The hardest design question on this case wasn't “how do we show the AI output?” — it was “what happens when the auditor disagrees?” That moment of disagreement is where an AI-augmented workflow either earns trust or loses it. Too frictionless, and overrides become thoughtless; too heavy, and auditors route around the tool. Putting a required justification on the override — and then telling the auditor where that note actually goes (the learning loop) — reframed it from compliance theater to collaboration with the model.