The Most Important Thing an AI Assistant Does Is Say 'I Don't Know'

Every vendor demo I have sat through follows the same arc. Someone types a question, the system thinks for a moment, and out comes a clean paragraph with two citations under it. The room nods. Somebody asks a second question, a harder one, and the same thing happens, and the room nods again.

Nobody in the room ever asks the third question — the one where the answer is not in the corpus at all.

I have started asking it, and I would encourage anyone evaluating one of these systems to make it the first thing they do rather than the last. Not because it is a trick. Because the way a system behaves when it does not know is the property that determines whether you can trust anything else it says.

The failure is quiet, which is the problem

A retrieval system that cannot find a relevant document has two options. It can say so, or it can answer from the general knowledge baked into the model.

The second option produces text that is fluent, confident, plausible in structure, and completely disconnected from your organization. It will describe how a hospital typically handles the thing you asked about. Sometimes that description will be right, which is worse, because it trains your staff to believe the next one.

Notice how different this is from an ordinary software failure. When a system is down, everybody knows. When search returns nothing, the user knows to go ask somebody. But an answer generated in the absence of a source looks exactly like an answer generated from a source. There is no visible difference at the point of use. The nurse reading it has no way to distinguish "this is your policy" from "this is what policies like this usually say."

This is what people are pointing at when they say hallucination, and I think the word does more harm than good. It suggests a model defect, which suggests the fix is a better model. Most of what I see in operational settings is not the model inventing things. It is the model answering the question it was asked with the only material it had, because retrieval came back empty and nothing in the system was configured to stop there.

That is a policy failure, not a model failure. And policy failures are fixable this quarter.

Abstention is a design decision, not an emergent property

Saying "I don't know" is not something a system does naturally. It has to be built, and it has to be built at a specific place in the pipeline.

The mechanics are less exotic than they sound. Retrieval returns candidate passages with some measure of how well they match. Somewhere there is a threshold. Below that threshold the system either declines to answer or it proceeds anyway. Where that threshold sits, and what happens below it, is a choice somebody made — and in a great many products, the choice was made implicitly by whoever wanted the demo to look good.

There is a real tension here and I do not want to pretend otherwise. Set the threshold high and the system refuses questions it could have answered, which is infuriating and kills adoption faster than a wrong answer does. Set it low and you get confident fiction. There is no universally correct setting, and anyone who tells you their number is right for your corpus has not seen your corpus.

What I would insist on is that the setting be visible and adjustable by you, that the behaviour below the threshold be an explicit refusal rather than a silent fallback, and that refusals be counted. A system that declines and logs the question is doing something useful with its failure. A system that declines and forgets has thrown away the most valuable thing it produced.

The refusal should not be a dead end

There is a version of this that overcorrects. A system that responds to anything difficult with a flat "I could not find an answer" is technically honest and practically useless, and staff will stop asking within a fortnight.

A good refusal does three things. It says plainly that it did not find an approved source. It shows what it did find, even though it judged the match too weak, because a human can often tell in two seconds whether the near-miss document is the right one. And it routes — names the person or the queue that handles this class of question, so the interaction ends with the user closer to an answer than they started.

The third part is where most implementations stop short, and it is not a technical limitation. It requires somebody to have written down who owns what, which brings you back to the unglamorous work that no software purchase avoids.

The refusals also have to go somewhere a human reads. Every declined question is a precisely located, demand-weighted gap in your documentation, described in the exact words a real person used. That is a content roadmap generated for free by the people who need the content. Solution Compass treats those unanswered questions as a first-class output rather than an error log, and I would push any vendor to show you where that list lives before you look at anything else in their product.

How to test it in a demo

You do not need a technical evaluation to check this. You need about ten minutes and a little discipline.

Bring five questions of your own. Make two of them answerable from documents you have supplied, so you can confirm the system works at all. Make one about something genuinely specific to your organization that you know is written down nowhere — the local escalation quirk on one unit, the reason a particular workflow was configured the way it was. Make one that sounds like your domain but is about a policy you have never had. And make one that is simply outside the scope entirely.

Then watch what comes back for the last three. You are looking for a refusal, and you are looking for the refusal to be specific rather than generic. If the system produces a confident paragraph about the policy you do not have, you have learned everything you need to know and you can stop the meeting early.

Be a little suspicious if the vendor asks to see your questions beforehand. There are legitimate reasons to ask, but the demo you want is the one nobody tuned for.

Why this is worth being stubborn about

Healthcare is unusual in how asymmetric the costs are here.

A system that says "I don't know" when it could have answered costs you a little time and some goodwill. Somebody asks a colleague, which is what they were doing before, and they grumble about the new tool. That is a recoverable disappointment.

A system that answers confidently from nothing costs you a decision made on false authority, by someone who had no way to know it was false, in an environment where the decision touches a patient or a regulator. That is not recoverable, and it will not be discovered at the moment it happens. It will be discovered later, by somebody with a clipboard.

Given that asymmetry I would take an annoying system over an agreeable one every time, and I would rather explain a refusal to a frustrated nurse than explain a fabrication to a surveyor.

So when the demo gets to the part where everything works, interrupt. Ask the thing the corpus cannot possibly know. The five seconds after that question is the only part of the meeting that tells you anything.