Can AI Chatbots Say "I Don't Know"? What OpenAI and Anthropic Found

Flat vector illustration of a thoughtful chatbot with a question mark, titled Can AI Chatbots Say 'I Don't Know'?

Yes — and the reason they so often invent answers instead is not that they are incapable of saying "I don't know." OpenAI's own researchers put it plainly in a September 2025 research post: "standard training and evaluation procedures reward guessing over acknowledging uncertainty." When a model is graded only on accuracy, declining to answer guarantees zero points while a lucky guess might score, so the models learn to guess. Anthropic's developer guidance answers from the other side: "Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information." The behavior is trainable — and you, as the person writing the prompt, can change it too.

Why the scoreboards teach models to guess

OpenAI illustrates the incentive with a multiple-choice analogy. If you do not know the answer but take a wild guess, you might get lucky. Leaving it blank guarantees a zero. Models face the same arithmetic: graded only on the percentage of questions they get exactly right, they are encouraged to guess rather than say "I don't know."

The research post uses a birthday example. Asked for someone's birthday, a model that guesses "September 10" has a 1-in-365 chance of being right. Saying "I don't know" guarantees zero points. Over thousands of test questions, the guessing model climbs the leaderboards while the careful model that admits uncertainty sinks.

OpenAI's Model Spec takes the opposite stance: it is "better to indicate uncertainty or ask for clarification than provide confident information that may be incorrect." Errors, the researchers write, are worse than abstentions. The paper behind the post — "Why Language Models Hallucinate" by Kalai, Nachum, Vempala, and Zhang — argues the fix is socio-technical: benchmark scoring has to stop penalizing abstention, because a dedicated hallucination test "has little effect against hundreds of traditional accuracy-based evals that penalize humility and reward guessing." If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.

The numbers: one test, two models, very different honesty

The GPT-5 system card includes a telling comparison on SimpleQA, an evaluation of short factual answers. The newer gpt-5-thinking-mini abstained on 52% of questions, answered 22% correctly, and got 26% wrong. The older o4-mini abstained on 1%, got 24% right — and got 75% wrong. The older model looked slightly better on pure accuracy but hallucinated nearly three times as often. Refusal behavior is not fixed by the architecture; it is set by training, and training can move it.

Two more findings from the same paper matter. First, hallucinations are not inevitable, because "language models can abstain when uncertain." Second, accuracy will never reach 100%, because "some real-world questions are inherently unanswerable" — information that does not exist, ambiguous questions, limits of the model itself. And counterintuitively, it can be easier for a small model to know its limits: a model that knows no Māori can simply decline a Māori question, while one that knows some Māori must judge its own confidence. Being calibrated, the authors note, takes far less computation than being accurate.

What the labs tell developers to do

Anthropic's "Reduce hallucinations" guide reads like a short manual for teaching a model to admit uncertainty:

  • Allow the model to say "I don't know." Explicit permission to admit uncertainty is the first technique listed — and the one Anthropic says "can drastically reduce false information."
  • Require direct quotes for long documents. Have the model extract word-for-word quotes before answering, grounding the response in the actual text.
  • Verify with citations. Make the model cite a quote for each claim, then check every claim against its quote — and retract anything it cannot support.
  • Advanced options: verify the model's reasoning steps, compare outputs across multiple runs, refine answers iteratively, or restrict the model to only the documents you provided.

The same techniques work when you are the one writing the prompt.

What you can change in the prompt yourself

Give explicit permission. Add one line: "Say 'I don't know' when you are not sure, rather than guessing." The permission changes the behavior because assistants are tuned to be helpful, and silence about uncertainty reads as a license to guess.

Demand the evidence, not just the answer. Ask for a quote or source for each factual claim, then ask the model to check every claim against its own quotes and drop the ones it cannot back up. This is Anthropic's citation check, and it works as a user prompt.

Use structured prompts for high-stakes questions. A 2025 study by Prateek Chhikara across nine large language models and three factual question-answering datasets found that adding "distractor" options — forcing the model to weigh alternatives instead of free-generating — produced relative accuracy improvements of up to 460% and cut calibration error by up to 90%. The study adds two warnings: large RLHF-tuned models can be miscalibrated on easier questions, and person-based questions (who someone is, dates, identity) remain a persistent weak spot. Verify those independently.

The honest limits

None of this eliminates invented answers. OpenAI is explicit that accuracy will never reach 100%. And grounding a model in retrieved documents is not a cure either. Stanford's RegLab tested three commercial legal-research AI tools that use retrieval — Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI — and found each hallucinated on 17% to 33% of preregistered queries, despite vendor claims of "hallucination-free" citations. The providers' claims, the researchers concluded, were overstated.

Researchers also disagree on where the fix belongs: train abstention into the model through better scoring and incentives, or handle it as a separate checking step outside the model. What both sides agree on is that the current default — guessing — is a product of how these systems are graded, not a law of nature.

The bottom line

The question is no longer whether a chatbot can be taught to say "I don't know." It can — the labs already teach it, and the numbers show the behavior moves. The more useful question for you is whether the product in front of you rewards honesty. Use the permission line, demand quotes, and check anything expensive. For facts that decide something — money, reputation, a legal filing — verify outside the chatbot, because the scoreboards that trained it still reward the confident guess.

Related: How AI Chatbots Generate Answers — and Why They Sometimes Invent Facts covers the prediction machinery this post builds on.

Sources

Comments

Popular posts from this blog

AI and Jobs: Which Tasks Change First, According to Researchers

What Marketing Tasks Will AI Replace Next — and Which Still Need a Human?

How AI Screens Your Resume Before a Human Sees It — and How to Write for It