Can AI Chatbots Say "I Don't Know"? What OpenAI and Anthropic Found
Yes — and the reason they so often invent answers instead is not that they are incapable of saying "I don't know." OpenAI's own researchers put it plainly in a September 2025 research post: "standard training and evaluation procedures reward guessing over acknowledging uncertainty." When a model is graded only on accuracy, declining to answer guarantees zero points while a lucky guess might score, so the models learn to guess. Anthropic's developer guidance answers from the other side: "Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information." The behavior is trainable — and you, as the person writing the prompt, can change it too. Why the scoreboards teach models to guess OpenAI illustrates the incentive with a multiple-choice analogy. If you do not know the answer but take a wild guess, you might get lucky. Leaving it blank guarantees a zero. Models face the same arithmetic: graded ...