Heuristics: Inappropriate high-confidence of "expert" decision-makers

Quite perversely, the more experience we acquire in decision-making the less accurately our mental models reflect how we actually make decisions.

Researchers in decision-making (Tversky, Kahnman, Slovic, Dawes and a host of others) studied expert decision-makers with the intention of improving diagnostic and financial decision making.

Financial decision making was researched, in part, because there is a wide body of non-volatile data that is readily available to the public. If you want to know the quarterly payroll and revenues of an obscure company in 1963 you can still find that data. That makes it possible to extract the data environment the decision maker was swimming in -after the fact- and review the results in real-time.

The hope was that insights gained by explicitly studying the thought processes of experts could be leveraged to other fields, like medicine, and could be taught to younger traders resulting in a much faster learning-curve. It was also hoped that computers could be programmed to make the same decisions stockbrokers made, but make them faster and more reliably (AI).

The earliest studies involved interviewing stock brokers in the top 5%. A typical high-end broker would wax eloquent about carefully evaluating twenty or more variables looking back five years or more. They said they evaluated the variables in highly non-linear ways. For example, growth in the number of employees was a positive when certain conditions were met but a negative when other conditions prevailed.

Young stockbrokers were also interviewed to define the starting point and get a picture of how much "learning" occurred. It was not surprising that novice stock pickers used simple, cook-book formulas. For example, a client might want to buy a position in a specific industry. The newbie would look at the candidate's Price/Earnings ratio less extraordinary flows, Revenue growth and future duration of Patent protections. The newbie would sort the candidates and present the client with his top three recommendations and tag "the best" choice.

That was the state-of-the-art when computers became widely available.

One of the first things those who studied decision making did was to "test" the model expert stockbrokers SAID they used against the data.

Performing linear regressions against their decisions and the data that was available at the time they made those decisions, it was determined that the decision-making process of "experts" was indistinguishable (given reasonable certainty bands) from the decision-making process of the newbies.

How can that be so?

The human brain is a computer with a very small buffer.

Experts used the same three, perhaps four variables that the newbies used. Then they used the remaining seventeen variables (remember, they said they used twenty or more) to CONFIRM their initial assessment.

At this point you should be thinking...Ah! This sounds like the First Impressions or Anchoring-and-Adjustment heuristic.

The balderdash about "...highly non-linear/configural..." arose from two sources.

More often than we care to admit, firms that appear great on paper falter and we do not get the financial return we predicted. Looking backwards, the stockbrokers would find some anomaly in the firm's statistics and create ad hoc "rules" to explain after-the-fact why that firm did not live up to their predictions.

The other origin of the "...highly non-linear/configural..." BS was that the experts always ignored statistics that contradicted their original assessment. But that caused cognitive dissonance so they eased the pain by rationalizing "...too many teraquarks per employe is usually a negative but this is such a great company and in this industry that, in this case, it is a positive..."

Most newbie stockbrokers were appropriately humble. They knew what they did not know. The "expert" stockbrokers were confident at levels far beyond what was appropriate for the actual processes he uses.

Do experienced stockbrokers add value?

One might think that an advisor who is grossly unaware of his actual decision making process and offers no measurable advantage over a newbie offers no value to the investor.

The advantage of an experienced advisor (financial, medical, etc.) is that lack of confidence is often very expensive.

Selling your stock holdings after the market pukes? Very expensive.

Waffling and not choosing a course-of-therapy when you have cancer? Very expensive.

The senior advisor is reassuring and helps you stay-the-course even as things get rocky.

The downside is that he is likely to keep you invested in buggy-whips when the less confident newbie would move you out.