August 2004

Medical research stats often wrong

It’s generally accepted that MDs should make their clinical decisions based on published scientific evidence. Yet, all too often, published findings are inaccurate, according to an article published in BMC Medical Research Methodology.
Researchers report that a large proportion of articles in top science and medical journals, including Nature and BMJ (formerly the British Medical Journal), contain statistical errors. At least some of these errors may have caused non-significant findings to be misrepresented as being significant. Fully, 38% of the Nature papers and 25% of the BMJ articles studied contained a minimum of one statistical error, according to Emili García-Berthou, a lecturer on biostatistics at University of Girona, Spain, and Carles Alcaraz.

“Our findings confirm that the quality of research and scientific papers needs improvement and should be more carefully checked and evaluated in these days of high publication pressure,” concluded the authors.

The errors seen could have been caused by transcription or typesetting errors, for example if a repeated zero was omitted. Alternatively, researchers may have rounded up figures incorrectly.

The researchers showed that some numbers, four and nine, were seen less often than would be expected at the end of a given test statistic or P-value, suggesting that researchers were rounding up numbers incorrectly, possibly so that they looked “neater.”

For example, study authors might round up 2.38 to 2.5 rather than 2.4, the report explained. “Although these kinds of errors may leave the conclusions of a study unchanged, they are indicative of poor practice,” stated the researchers. “Our concern is that these kinds of errors are probably present in all numerical results and all steps of scientific research, with potentially important practical consequences.”

The researchers suggested that one way to minimize the effect of these errors would be for published authors to make their raw data freely available on the Internet. This would allow other researchers to check for themselves whether the results of the study are correct and the conclusions justified. “Also, fraud and sloppiness may be more easily detected,” they said.

SOURCE: “Incongruence between test statistics and P values in medical papers,” by

Emili García-Berthou and Carles Alcaraz, BMC Medical Research Methodology, May 28, 2004.