False Binary Choices: Part 2

Spurious detail creates unwarranted confidence

  "Sam has a quiet demeanor and was a good listener. He calmly folds his hands on his desk in front of him while listening. He wears eye-glasses with extra-large lenses and thick, black rims. At home, he favors Argyle socks and old, comfortable cardigans. Sam cares about the environment. He once sold a cat-carrier to his next-door neighbor for $20 and laboriously filled out a bill-of-sale."

The previous paragraph is the kind of "description" authors insert right before or after introducing a new character. 

The characteristics in the paragraph convey NO INFORMATION about Sam's profession. Regardless, you will have much more confidence if I inform you that Sam is a librarian or an accountant. EVEN THOUGH he could just as easily be a cop or a brick-layer or a hit-man for the mob.

Bayes' theorem informs us that each incremental piece of information contributes decreasing amounts of "confidence" (as a mathematician would define it) to a body of information or its ability to support a hypothesis.

Bayes' theorem is powerful because the human mind does not process that way. If the additional piece of data supports our hypothesis it is grossly over-weighted and if the data contradicts our hypothesis then it is ignored.

The author of the article referenced yesterday lamented the "certainty" and "confidence" created by false, binary choices even though the peer-reviewed literature is filled with cautions about misplaced confidence in sophisticated analysis.

Case One: A large automaker purchased sophisticated software in the mid-1980s to process strain-gage data. The software offered several calibration options, from "linear" to fit-with 5th-order-polynomial.

The engineer in charge decided that 5th-order-polynomials HAD to be better than simple linear (which is what the company had been using, successfully, for 40 years).

After six months of data collection and processing, somebody printed out a one of the calibration curves and looked at it. It looked like a snake with several concave-ups and concave-downs. There was no plausible, physical reasons for that degree of curvature.

The program was restarted and suddenly became a very expensive "crash program" that ran on overtime and caffeine because it was six months behind.

Case Two: A major newspaper wrote a huge series on various school districts in the Detroit area and their results on the Michigan Educational Assessment Tests. They grade the school districts based on whether they outperformed the computer predictions or under-performed them.

At the end of the article they listed the variables that were used. The weightings on the variables were derived by "regressing" data from the previous year against student scores.

A partial list of the variables:

• Percent single-parent homes
• Average income
• Percent rental housing
• Percent subsidized lunches
• Percent students who identified as "minorities"
• Drop-out rate 
• Percent students taking College Entrance tests
  
• Crime rate
• Funding per-student
• Teacher-student ratio
• Percent of teachers with advanced degrees
  

If you look at those variables you might assume that they VARY together. That is, they are manifestations of a common cause(s).

One quirk of "regression" or curve-fit software is that factors with low correlation with the results will often have HUGE weighting factors while factors with high correlation will have more modest weights. That means that noise in the factors with less correlation will have a disproportionate impact on predictions.

There can also be "perverse" weights. For example, a very high drop-out rate will remove the least capable students and can be expected to raise standardized test scores relative to the actual population of students who started their educational career in that school-district. The percent students who took college entrance exams (ACT, SAT) can function in the same way. All the administration must do to "outperform relative to prediction" is to tell their least motivated students that the test is for college entrance and they are excused not attend school that day. 

Case Three: Robyn Dawes was a professor at Carnegie-Melon University and he wrote a brilliant paper on "The Robust Beauty of Improper Linear Models in Decision Making" (behind paywalls)

He evaluates the results of computer models where the weightings (which were between the values of -1 and 1 and listed with two-digits of precision) were replaced with the equivalent of -1, 0, 1.

Since multiplying any variable with "0" makes it disappear, the model becomes binary...either -1 or 1 being assigned to each variable. If it is important then it either pushes or pulls.

Professor attributes the concept to Benjamin Franklin who recommended that decision makers take a sheet of paper and draw a line down the center. On one side of the line write down all of the reasons why you should decide in favor and on the other side write down all of the reasons why you should decide against.

Whichever column is longer wins.

So Robyn Dawes wrote a paper suggesting that "Improper Linear Models", i.e. binary models, can run neck-and-neck with far more sophisticated, more expensive, slower models.

A question

So who has the unwarranted confidence? The people who believe in "False Binary Choices" or the people who navigate in a foggy landscape cluttered with spurious data trying to decide between a continuum of subtle, shifting shades of gray?