How vulnerable is the field of cognitive neuroscience to bias?

That's the opening sentence in an abstract by Joshua Carp, who will be presenting tomorrow in Slide Session 3 at the 2012 CNS Meeting in Chicago. The question caught my eye in light of the Psych Science paper on False-Positive Psychology ["undisclosed flexibility in data collection and analysis allows presenting anything as significant"] and the recent blog post by Dr Daniel Bor on The dilemma of weak neuroimaging papers.

Slide Session 3

Sunday, April 1, 10:00 am - 12:00 pm, Red Lacquer Room

Estimating the analytic flexibility of functional neuroimaging: Implications for uncertainty and bias in cognitive neuroscience

Joshua Carp; University of Michigan

How vulnerable is the field of cognitive neuroscience to bias? According to a recent mathematical model, the potential for scientific bias increases with the flexibility of analytic modes. In other words, the greater the range of acceptable analysis strategies, the greater the likelihood that published research findings are false. Thus, the present study sought to empirically estimate the analytic flexibility of fMRI research. We identified five pre-processing decisions and five modeling decisions for which two or more analysis strategies are commonly used in the research literature. By crossing each of these strategies and decisions, we identified 4,608 unique analysis pipelines. Next, we applied each of these pipelines to a previously published fMRI study of novelty detection in an auditory oddball task. We found that activation estimates were highly dependent on methodological decisions: contrasts that yielded significant positive activation under one pipeline were associated with non-significant positive activation or even with negative activation under other pipelines. Some analysis decisions contributed more to this variability more than others, and each decision exerted a unique pattern of variability across the brain. The effects of a given decision also varied across contrasts, subjects, and other analysis parameters. In sum, we found considerable quantitative and qualitative variability across analysis pipelines, suggesting that the results of cognitive neuroimaging experiments may be more uncertain than they seem. Indeed, given a supercomputer, a sufficiently motivated analyst might observe almost any imaginable pattern of results.

Reference

Simmons JP, Nelson LD, Simonsohn U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 22:1359-66.