Noise #2: Each Case is Unique, and One-Off Cases

In their book titled Noise, the authors clarify that they understand why real-world decisions are so “noisy”:

“Judgment is difficult because the world is a complicated, uncertain place.”

However, they continue:

“There is a limit to how much disagreement is admissible.”

Taken too far:

“System noise is inconsistency, and inconsistency damages the credibility of the system.”

 

Take the fact that different judges give different sentences for the same crime:

“This variability cannot be fair. A defendant’s sentence should not depend on which judge the case happens to be assigned to.”

Attempts to fix this by issuing guidelines for judges, which restrict the variation among judges, have been resisted by judges. Why?

“After all, each case is unique, isn’t it?”

Yes, the authors concede, sometimes the need for discretion is important. Their quarrel though is with all the too many cases where “variability is undesirable”. Like different premium quotes by insurance companies for pretty much the same risk (e.g. your life/car insurance quote).

 

A common argument given is that the errors cancel out on average, so such noise shouldn’t be a big deal. Wrong, say the authors. By charging too high a premium, the insurance company loses a client. But by charging too low a premium, the company’s profits suffer (and it exposes itself to greater risks).

“In noisy systems, errors do not cancel out. They add up.”

 

While it may be possible to spot the noise when the same/similar situation occurs repeatedly, what about one-off cases? By definition, one can’t have noise in those, right (there’s only one instance of it)? The authors have an ingenious answer: “If we think counterfactually”, we can identify if noise is part of even that one-off decision. How’s that? Look at all the factors that went into the decision (e.g. a decision to acquire a company, or to select a kid for a sports team), they say. Is it possible that a different person (manager, coach etc) could have made a different decision? Due to different inputs or background or life history? If the answer to those questions was Yes, then there must be noise even in that one-off decision.

 

Fine, but that sounds like an academic point, you say. Aha, but the authors are driving at a different point here:

“If singular decisions are as noisy as recurrent ones, then the strategies that reduce noise in recurrent decisions should also improve the quality of singular decisions.”

 

With that, we’ll next look at the causes of noise, and then possible solutions to reduce it.