Origins of Excessive Optimism

I think it is important to understand the origins of "excessive optimism" if one hopes to armor one's self against poor decisions making.

Iterative discounting
One source of excessive optimism is rooted in the limitations of how human minds process data. Frankly, we are one-holers with limited buffer.

The way we process complex problems it to solve them piece-wise and iteratively. It is like combing the snarls out of your daughter's hair. You isolate a small hank and comb out the first half-inch, then inch, then inch-and-a-half.... Then grab another hank.

The error creeps in when the decision maker applies the "discounting factor" to the negative possibilities.

Suppose the possibilities of a negative outcome is 40%. If that discounting rate is applied iteratively three times then the apparent probability of  a negative outcome sinks ot 6.5%.  If the hank-of-hair is long and is combed five times then the perceived risk of a negative outcome sinks to 1.0%...a long way from the original assessment of 40%, nearly a flip of the coin.

One countermeasure for this is to produce a formal, visual representation of the issue, the enablers and the primary sources of risk. Draw a flow-chart and explicitly assign risks to each major factor and let a computer modeler grind out the risk of the plan in-total. Then decide of the reward warrants the risks.

Using comfort as a proxy for risk
Suppose you had been given an assignment you never had before. For the sake of the absurd, suppose you had to teach Sunday School to a bunch of fifteen-year-old Preacher's kids.

The first Sunday you walked into class you were probably walking on pins-and-needles and were very uncertain of how it would go.

By the last Sunday of the year you were probably very at ease.

However, the chances of your stepping in a mess is probably just as high on the last Sunday as it was the first Sunday. What changed was your comfort level and hence your perception.

That is one of the risks of excessive planning. Comfort grows far faster than risk diminishes.

Perhaps that is why we date when seeking a mate.  We are growing comfort, not reducing risk. It is likely that most people had enough information by the fifth date to know if their BF/GF would make a good mate. Subsequent dates were for comfort and confirmation.

One countermeasure is to boil in "OH SHIT!" events. Can the situation and the dynamics survive one of you getting cancer or a special-needs kid or the loss of a job or a cross-country assignment? Let's be honest, noise happens and it is not difficult to game possible situations of a magnitude that are likely to happen in the first five years.