Posts

Showing posts with the label AI chatbots AI hallucinations LLM training ChatGPT Claude AI honesty prompt engineering artificial intelligence

Can AI Chatbots Say "I Don't Know"? What OpenAI and Anthropic Found

Image
Yes — and the reason they so often invent answers instead is not that they are incapable of saying "I don't know." OpenAI's own researchers put it plainly in a September 2025 research post: "standard training and evaluation procedures reward guessing over acknowledging uncertainty." When a model is graded only on accuracy, declining to answer guarantees zero points while a lucky guess might score, so the models learn to guess. Anthropic's developer guidance answers from the other side: "Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information." The behavior is trainable — and you, as the person writing the prompt, can change it too. Why the scoreboards teach models to guess OpenAI illustrates the incentive with a multiple-choice analogy. If you do not know the answer but take a wild guess, you might get lucky. Leaving it blank guarantees a zero. Models face the same arithmetic: graded ...