SEAL OF DISAPPROVAL

How Much of the Neuroimaging Literature Should We Discard?

Since sub-optimally designed and analyzed fMRI studies continue to influence the field, there should be some mechanism for identifying discredited publications. This discussion was initiated by Professor Dorothy Bishop's critical analysis of a flawed paper. In response, Dr Daniel Bor wrote a thoughtful post on The dilemma of weak neuroimaging papers. He covered corrected and uncorrected statistics, a culture of sloppy neuroimaging publications, and whether a bad paper can be harmful or should be retracted.


Does Discarding Mean Retraction?

By "discarding" I meant disregarding the results from flawed articles, not retracting them from the literature entirely. This point was misunderstood by some. It seems that all commenters on Bor's post agreed that retraction is problematic.

Bor: "...it’s almost certainly impractical to retract these papers, en masse."

Neuroskeptic: "Retracting them all… never going to happen, but even it did, I don’t think it would help at all. Much better would be for readers to educate themselves or be educated to the point where they know how to spot sloppy stats."

Bishop: "I don’t think it’s realistic to expect a retraction, and certainly not just on the basis of reporting of uncorrected statistics. I think we should disregard the results of this study because of the constellation of methodological problems..."

Poldrack: "I think retraction of papers that report uncorrected statistics is a bit much to ask for; after all, most of the results that were published in the days before rigid statistical corrections were common have turned out to replicate, and indeed large-scale meta-analyses have shown a good degree of consistency..."

This is not a new debate. In cellular and molecular biology, there have been cases of technical artifacts which resulted in failures to replicate. Retraction Watch, your authoritative source for all retraction news in the scientific literature, discussed this issue at length last year (e.g., So when is a retraction warranted? The long and winding road to publishing a failure to replicate). In a subsequent post, Ivan Oransky concluded:

So to get back to those Retraction Watch threads to which we referred: Yes, failure to replicate seems a good reason to retract. And notices should explain what went wrong. Allowing authors to get away with failing to do so, in some well-intentioned but misguided attempt to lower the barrier to retractions — as the Journal of Neuroscience does, for example — is part of why some people seem to think that retractions mean fraud.

This prompted a response from Isis the Scientist, who asked What Warrants a Retraction?

This story and controversy is still evolving. To retract the paper now, without evidence of overt fraud or negligence, will mark it with the scarlet watermark of fraud. Because a retraction is the foremost boner-killer of science, the retraction is a weapon to be wielded carefully. It should be used in cases of fraud and negligence, but not in cases where there remains active debate or inconclusive evidence.

Even then, there is value to leaving errors and conflict in the literature and there must be a better tool than the retraction watermark...

...but no one seems to know what that better tool is. Until now.



Introducing STAMPS OF DISAPPROVAL, by Heather K. Phillips

Gone are the days of tearing work from the wall. These days, disapproval often takes the form of ambiguous encouragements. Put the language of critique in your hands with this series of 12 rubber stamps. Each stamp bears of fragment of abridged feedback associated with critique.

Now available for purchase at Schooled

But seriously, improvements in how the field corrects itself will require "a structured information overlay for all academic papers," according to Ben Goldacre. Databases of failures to replicate and post-publication assessment are needed. Some scientists would like to overturn the current system of peer review entirely.

Or maybe we can incorporate a creative system that uses the STAMPS OF DISAPPROVAL...