Monday, December 6, 2021

Heuristic of the week: Insensitivity to predictability

 

Some phenomena are unpredictable but that does not prevent most people from parsing small samples of data and making sweeping predictions.

Consider the amount of data actually embedded in the final score of a football game: The lead may have seesawed back and forth several times and yet the team that was ahead when the clock ran out is considered the "superior" team.

Let us hypothesis that there is no significant differences between the approximately 100 Division I football teams in the US. A middle-of-the-road team could conceivably win all of its conference games by virtue of a lucky clock. 

If the typical "conference season" is eight games long then there is one-chance-in-256 of that happening. Stated another way one team (of the 100 Div I teams) will stumble into a perfect conference record every three years if our hypothesis is true, and yet every howling fan will believe their team is the G.O.A.T.

Humans are inherently unpredictable

Another venue where there is a high degree of unpredictability is in choosing employees or mates.

Visualize a school principal popping in to evaluate a student teacher. He observes for fifteen minutes and makes a hiring decision based on those observations.

One teacher's classroom is a trainwreck. It smells like vomit. The kids are totally out-of-control. The plan is in shambles and none of the "curriculum" for the grade-level is delivered.

A second teacher's classroom is the model of decorum and order. The children are happy and engaged. The material delivered is exactly where the kids should be for the time-of-year.

Who gets the job offer?

What if I told you Teacher One and Teacher Two were the same teacher but the observations were an hour apart...Teacher Two was observed before the student projectile vomited and Teacher One was the same teacher observed shortly after the student puked.

Summary

Some phenomena are inherently unpredictable.

Additional data might not be justified on the basis of cost and the incremental predictability it could bring to a decision.

When making decisions about situations that are inherently unpredictable, resources that are invested in collecting reams of data (which may not be analyzed or reviewed) might better be invested in back-up plans or plans to mitigate risk on-the-fly.

In the example of the student teacher: If he proves inadequate at maintaining classroom control then freeing up a teacher who excels at classroom control can observe him and point out tactical errors. Simple things like turning his back on the students or dithering at the start of the instruction session rather than immediately plunging into the work-at-hand.

Bonus Link

2 comments:

  1. How many of those unpredictable situations are unpredictable because of outside factors?
    For example, the puking student could have gotten sick from cafeteria food.
    The classic example of this is farming - the farmer can't control the weather and the weather has a huge impact on the crop.

    ReplyDelete
    Replies
    1. Most of them, probably.

      The opponent tries to have much to say about the outcome of the game.

      "Outside factors" is another way of saying "an input I do not have control over".

      Delete

Readers who are willing to comment make this a better blog. Civil dialog is a valuable thing.