Errors in Publications

Joe Daigle wrote an interesting article on the issues found when someone tries to replicate a research publication:

“A shocking fraction of published research in many fields, including medicine and psychology, is flatly wrong—the results of the studies can’t be obtained in the same way again, and the conclusions don’t hold up to further investigation.”

 

He then turns to a field which should be immune to this problem – maths:

“In experimental sciences, the experiment is the “real work” and the paper is just a description of it. But in math, the paper, itself, is the “real work”.”

Unlike other fields, where it takes money to try and replicate an experiment, one can just go over the other guy’s proof in maths (and spot errors), right? Nope:

“It’s reasonably well-known among mathematicians that published math papers are full of errors. Many of them are eventually fixed, and most of the errors are in a deep sense “unimportant” mistakes. But the frequency with which proof formalization efforts find flaws in widely-accepted proofs suggests that there are plenty more errors in published papers that no one has noticed.”

How can this be?

“As our mathematics gets more advanced and our results get more complicated, this replication process becomes harder: it takes more time, knowledge, and expertise to understand a single paper.”

 

Fortunately, things aren’t as bad as it sounds, at least in maths:

“Many papers have errors, yes—but our major results generally hold up, even when the intermediate steps are wrong! Our errors can usually be fixed without really changing our conclusions.”

This seems to happen because, for reasons unexplained, “people’s intuition about what’s true is mysteriously really good.”.

 

But that bit about intuition-being-a-good-guide-to-truth holds only in maths, not the other fields. In other fields, esp. psychology, one’s intuitions easily become biases leading to you-conclude-what-you-believe errors.

 

The problem of unable-to-replicate thus seems to be severe in the behavioral fields. Though the hard sciences have a different problem – the equipment, money, time, and even luck (e.g. observing a phenomenon in space) are so costly/long that they may be very hard to try and replicate.

 

In maths, the more unintuitive the “proof” or the longer a hypothesis has been unproved, the more the proof is scrutinized. That acts as a good check against strange (and thus headline grabbing) proofs don’t turn out to be wrong.

 

A lot of food for thought there.