How Appropriate is Absolute Certainty?

We all know somebody who is most adamantly convincing when they are least likely to be right.  It is as if they are over-compensating for their shaky logic with overbearing persuasiveness.

But how accurate is "absolute certainty" amongst the general run of the population.

Bottom Line

The bottom line is that most people are very poorly calibrated with regard to certainty.

It is a safe bet that there is a 20% chance of error when anybody claims "absolute certainty".

The research

A test was devised with questions like:

What is the country of origin for the White Potato
A.) Iraq
B.) Ireland
C.) France
D.) Peru

-and-

What is your confidence (certainty) that you chose the correct answer?
A.) 50%
B.) 90%
C.) 99%
D.) 99.99%

The test subjects were graduate students (in Psychology) at a major university.  The unspoken assumption is that these grad students needed to have some understanding of statistics to perform meaningful research and write a viable thesis.

The questions that the students self-assessed at 99.99% certain (that is, one wrong answer in every 10,000 questions) were compiled and 80% of them were found to be correct.  That is, 2,000 wrong answers in every 10,000 questions.

Take Two

Perhaps the issue was understanding of what "99.99% certain" really means?

A one hour class session was dedicated to explaining to a fresh group of subjects exactly what 99.99% certainty means.  That is, if you answered 10,000 questions and were 99.99% certain your answers were correct, then only one of those 10,000 questions would have an incorrect answer.

It was also made clear that most people are mis-calibrated and are excessively optimistic in the correctness of their answers.

A test similar to the test in Take One was administered.

Of the questions that were identified as being answered with 99.99% certainty, about 82% were found to be correct.  That is, the one hour lecture did not improve the performance by any significant amount.

Take Four

Two weeks of class time were dedicated to explaining the concept of 99.99% certainty.  Trial tests were run.  Results were fed back to the class.

At the end of two weeks of intensive training to graduate students at one of the top universities in the country the baseline test was administered.  Of the subset identified as 99.99% certainty about 95% were found to be correct.  That is, instead of one wrong answer in every 10,000 questions there were 500 wrong answers for every 10,000 questions.  Much better than the first three iterations but still far short of good calibration.


These results have been replicated many times.

Conclusions:

• Even "really smart" people are mis-calibrated with respect to "absolute certainty".
• It is reasonable to assume that mis-calibration with regard to "absolute certainty" is the human condition
• Mis-calibration is difficult to eradicate
• The human brain seems to round 80% certainty up to 99.99% certainty with great reliability
• When lacking hard, numerical data; it is safe to assume that anecdotal "absolute certainty" involves 20% fall-out.