Human beings are almost universally poorly calibrated with regard to "Certainty".
This poor calibration is most pronounced at the extremes, 100% Certain and 0% Certain.
Furthermore, this poor calibration is highly resistant to mitigation.
The scientific literature would be funny if it were not so sad.
A typical peer-reviewed paper describes test subjects who are in their second year of graduate study.
They are given multiple choice tests of "General Knowledge" content. A typical question might be:
The common, white potato is native to what country?
- A. Peru
- B. Ireland
- C. France
- D. Nepal
The subjects are asked to answer the question to the best of their ability and then to rank their "certainty" that they answered correctly. Some tests offered 0%, 25%, 50%, 75% and 100%. Other tests offered 90%, 99%, 99.9% and 99.99% for choices of certainty.
After scoring the tests and establishing the graduate student's dismal calibration, the authors used a one hour class session to discussing what certainty is and hammered the point that 99.99% certainty meant that 10,000 questions so labeled would only have ONE wrong and 99.9% meant one-in-one-thousand would be incorrect.
They re-ran the test (using different questions) and the results were identical at the highest confidence levels.
Graduate students. Who needed to use statistics to prove their thesis.
The authors gave the graduate students a WEEK LONG seminar in the meaning of confidence (in this context) and re-ran the test. The results on the high-end twitched the very tiniest amount.
0% Confidence
One would have expected that nobody would have labeled any of their answer 0% confidence as a random guess would have a 25% chance of being correct.
Nobody chose this level in the tests ran after the one-hour seminar.
Caveat Emptor
These studies were replicated at multiple, competitive-admission graduate schools.
These were not artifacts of small numbers. Many hundreds of students were involved in each replication.
Since the certainty of "experts" is almost universally poorly-calibrated and is so highly resistant to attempts to improve that calibration, it is incumbent upon the users of "expert opinion" to automatically substitute "80% certain" every time they hear the expert say "I am absolutely certain".
When an 'expert' is 100% certain, he's 80% likely to be right. When Fauxchi says he's 'absolutely certain', he is 100% certain that he is giving you the answer currently needed to cover his @$$, and you can be 100% certain that he's blowing smoke up yours.
ReplyDeleteWhat is useful about this research is it gives you some solid estimates to hang your hat on.
DeleteIf somebody says things are "50:50" odds are that their estimate is close.
If they say they are 100% confident than it is good to know that there is data that supports constructing contingency plans based on a 20% chance they are wrong.
When ANY scientist says they are 100% certain, they are either an idiot or a lying, and are probably trying to sell something...
ReplyDeleteAh.., but uncertainty makes people queasy. How many times has science qualified knowledge and it has been rejected? Look how the climate crisis has been exploited because "the science is uncertain", even though there is consensus among scientists.
ReplyDeleteThe universe has randomness built into it but there are lots going around saying, "everything happens for a reason."
"...uncertainty makes people queasy..." is the crux of the issue.
DeleteCognitive dissonance at its finest.
Most people navigate through life at a binary level. "Do I buy a truck this year or not?" "Do I buy and F-150 or Silverado?" "Do I get the gas or diesel engine?" and so on.
Cumulatively, the the uncertainty snowballs but if you treat it as sets of binary decisions where 80% certainty slides to 100% then you can arrive at your final decision with no dissonance.