Wrong AI Advice Cuts Accuracy, Doubles User Confidence
20 Jul 2026
The Study
Researchers from three French and Italian universities — Valerio Capraro, Chiara Marcoccia, and Walter Quattrociocchi — set out to test what happens when people receive AI advice that's deliberately wrong. They used questions about visual details from films, a domain where AI models like the one tested, Step 3.5 Flash, typically fail. The model was usually wrong on the test questions used in the study.
The results were stark. Without AI advice, participants said 'I don't know' 44% of the time; with AI advice available, that fell to just 3%. Accuracy dropped from 27% to 9%. Meanwhile, confidence rose from 30% to 76% — meaning people became roughly three times worse in accuracy while feeling more than twice as confident.
As Capraro put it: 'People became much worse, the accuracy was only one third, but they were twice as confident.'
Why 'I Don't Know' Matters
The researchers emphasize that the erosion of epistemic humility — the willingness to admit uncertainty — is itself a risk, independent of the accuracy drop. Capraro noted: 'For humans, the capacity to say I don't know is very important because it represents the recognition of the limits of our own knowledge.'
The report flags a related concern: children developing alongside AI systems before building critical thinking skills could face compounded risks. This aligns with Common Sense Media's separate assessment, which flagged Google's AI search design as an 'unacceptable risk' for students.
Can Incentives Fix It?
The study also tested whether monetary incentives could counteract the effect. They helped — but only modestly. Willingness to admit ignorance rose from 3% to 8%, and accuracy improved from 9% to 16%. Both remained far below the no-AI baseline, suggesting the overconfidence effect is not easily undone by adding stakes alone.
This finding echoes a broader pattern researchers have been tracking. Earlier this year, Wharton researchers coined the term 'cognitive surrender' to describe how people accept incorrect AI answers roughly 80% of the time.
What's Missing
The report notes several open questions: the exact sample size, participant demographics, and study design details aren't specified, nor is it clear whether findings extend beyond film-trivia questions to domains like coding, medical, or financial advice. The publication venue, peer-review status, and date of the study also aren't mentioned, and there's no data on whether other AI models would produce similar effects.
Why Founders Should Care
For founders building AI-advice products, this study suggests a few things are likely — though not certain, given the narrow domain tested.
- Users may become more confident even as accuracy declines, which could increase the odds that AI-assisted products generate misplaced trust rather than better decisions. This may be a strong argument for building explicit uncertainty-signaling into AI UX — flagging when a model is unsure or historically weak in a given area.
- Adding accountability or stakes (like the monetary incentives tested) appears to help only partially, implying that UX-level nudges alone probably won't fully solve overconfidence — founders may need to combine incentive design with clearer uncertainty framing.
- The 'cognitive surrender' concept suggests this risk could be broader than film trivia, potentially affecting any AI-advice product, though the report doesn't confirm this generalizes.
- Edtech founders in particular may want to weigh extra safeguards, given Capraro's concern about children and Common Sense Media's stance on AI search tools for students.
- Given growing scrutiny from groups like Common Sense Media, founders shipping AI search or advice features should probably expect increasing reputational and regulatory attention to how these tools shape user trust and confidence.
The Bottom Line
The core finding — accuracy dropping to roughly a third while confidence nearly doubled — is a meaningful signal that AI advice, even when wrong, can reshape human judgment in ways that are hard to reverse with incentives alone. For founders building products where AI offers advice or answers, the study is a reminder that designing for calibrated trust, not just engagement, may matter as much as model accuracy itself.