Public Service Announcement

*** THIS IS A PUBLIC SERVICE ANNOUNCEMENT ***

There is still time to re-analyze your data correctly, resulting in an accurate abstract submission to the 15th Annual Meeting of the Organization for Human Brain Mapping.

But hurry! This is a limited time offer!!

Deadline is January 11th, 2009 at 11:59 PM CST USA

Click here to submit an abstract

NOTE: all abstract submissions reviewed by Ed Vul.

Voodoo Correlations in Social Neuroscience

The end of 2008 brought us the tabloid headline, Scan Scandal Hits Social Neuroscience. As initially reported by Mind Hacks, a new "bombshell of a paper" (Vul et al., 2009) questioned the implausibly high correlations observed in some fMRI studies in Social Neuroscience. A new look at the analytic methods revealed that over half of the sampled papers used faulty techniques to obtain their results.

Edward Vul, the first author, deserves a tremendous amount of credit (and a round of applause) for writing and publishing such a critical paper under his own name [unlike all those cowardly pseudonymous bloggers who shall go unnamed here]. He's a graduate student in Nancy Kanwisher's Lab at MIT. Dr. Kanwisher¹ is best known for her work on the fusiform face area.

Credit (of course) is also due to the other authors of the paper (Christine Harris, Piotr Winkielman, and Harold Pashler), who are at the University of California, San Diego. So without further ado, let us begin.

A Puzzle: Remarkably High Correlations in Social Neuroscience

Vul et al. start with the observation that the new field of Social Neuroscience (or Social Cognitive Neuroscience) has garnered a great deal of attention and funding in its brief existence. Many high-profile neuroimaging articles have been published in Science, Nature, and Neuron, and have received widespread coverage in the popular press. However, all may not be rosy in paradise:²

Eisenberger, Lieberman, and Williams (2003), writing in Science, described a game they created to expose individuals to social rejection in the laboratory. The authors measured the brain activity in 13 individuals at the same time as the actual rejection took place, and later obtained a self-report measure of how much distress the subject had experienced. Distress was correlated at r=.88 with activity in the anterior cingulate cortex (ACC).

In another Science paper, Singer et al. (2004) found that the magnitude of differential activation within the ACC and left insula induced by an empathy-related manipulation was correlated between .52 and .72 with two scales of emotional empathy (the Empathic Concern Scale of Davis, and the Balanced Emotional Empathy Scale of Mehrabian).

Why is a correlation of r=.88 with 13 subjects considered "remarkably high"? For starters, it exceeds the reliability of the hemodynamic and behavioral (social, emotional, personality) measurements:

The problem is this: It is a statistical fact... that the strength of the correlation observed between measures A and B reflects not only the strength of the relationship between the traits underlying A and B), but also the reliability of the measures of A and B.

Evidence from the existing literature suggests the test-retest reliability of personality rating scales to be .7-.8 at best, and a reliability no higher than .7 for the BOLD (Blood-Oxygen-Level Dependent) signal. If each of these measures was [impossibly] perfect, then the highest possible correlation would be sqrt(.8 * .7), or .74.

This observation prompted the authors to conduct a meta-analysis of the literature. They identified 54 papers that met their criteria for fMRI studies reporting correlations between the BOLD response in a particular brain region and some social/emotional/personality measure. In most cases, the Methods sections did not provide enough detail about the statistical procedures used to obtain these correlations. Therefore, a questionnaire was devised and sent to the corresponding authors of all 54 papers:

APPENDIX 1: fMRI Survey Question Text

Would you please be so kind as to answer a few very quick questions about the analysis that produced, i.e., the correlations on page XX. We expect this will just take you a minute or two at most.

To make this as quick as possible, we have framed these as multiple choice questions and listed the more common analysis procedures as options, but if you did something different, we'd be obliged if you would describe what you actually did.

The data plotted reflect the percent signal change or difference in parameter estimates (according to some contrast) of...

1. ...the average of a number of voxels.
2. ...one peak voxel that was most significant according to some functional measure.
3. ...something else?

etc.....

Thank you very much for giving us this information so that we can describe your study accurately in our review.

They received 51 replies. Did these authors suspect the final product could put some of their publications in such a negative light?

SpongeBob: What if Squidward’s right? What if the award is a phony? Does this mean my whole body of work is meaningless?

After providing a nice overview of fMRI analysis procedures (beginning on page 6 of the preprint), Vul et al. present the results of the survey, and then explain the problems associated with the use of non-independent analysis methods.

...23 [papers] reported a correlation between behavior and one peak voxel; 29 reported the mean of a number of voxels. ... Of the 45 studies that used functional constraints to choose voxels (either for averaging, or for finding the ‘peak’ voxel), 10 said they used functional measures defined within a given subject, 28 used the across-subject correlation to find voxels, and 7 did something else. All of the studies using functional constraints used the same data to select voxels, and then to measure the correlation. Notably, 54% of the surveyed studies selected voxels based on a correlation with the behavioral individual-differences measure, and then used those same data to compute a correlation within that subset of voxels.

Therefore, for these 28 papers, voxels were selected because they correlated highly with the behavioral measure of interest. Using simulations, Vul et al. demonstrate that this glaring "non-independence error" can produce significant correlations out of noise!

This analysis distorts the results by selecting noise exhibiting the effect being searched for, and any measures obtained from such a non-independent analysis are biased and untrustworthy (for a formal discussion see Vul & Kanwisher, in press, PDF).

And the problem is magnified in correlations that used activity in one peak voxel (out of a grand total of between 40,000 and 500,000 voxels in the entire brain) instead of a cluster of voxels that passed a statistical threshold. Papers that used non-independent analyses were much more likely to report implausibly high correlations, as illustrated in the figure below.


Figure 5 (Vul et al., 2009). The histogram of the correlations values from the studies we surveyed, color-coded by whether or not the article used non-independent analyses. Correlations coded in green correspond to those that were achieved with independent analyses, avoiding the bias described in this paper. However, those in red correspond to the 54% of articles surveyed that reported conducting non-independent analyses – these correlation values are certain to be inflated. Entries in orange arise from papers whose authors chose not to respond to our survey.

Not so coincidentally, some of these same papers have been flagged (or flogged) in this very blog. The Neurocritic's very first post 2.94 yrs ago, Men are Torturers, Women are Nurturers..., complained about the overblown conclusions and misleading press coverage of a particular paper (Singer et al., 2006), as well as its methodology:

And don't get me started on their methodology -- a priori regions of interest (ROIs) for pain-related empathy in fronto-insular cortex and anterior cingulate cortex (like the relationship between those brain regions and "pain-related empathy" are well-established!) -- and on their pink-and-blue color-coded tables!

Not necessarily the most sophisticated deconstruction of analytic techniques, but it was the first...and it did question how the regions of interest were selected. And of course how the data were interpreted and presented in the press.

SUMMARY from The Neurocritic : Ummm, it's nice they can generalize from 16 male undergrads to the evolution of sex differences that are universally valid in all societies.

As you can tell, this one really bothers me...

And what are the conclusions of Vul et al.?

To sum up, then, we are led to conclude that a disturbingly large, and quite prominent, segment of social neuroscience research is using seriously defective research methods and producing a profusion of numbers that should not be believed.

Finally, they call upon the authors to re-analyze their data and correct the scientific record.

Footnotes

¹ Kanwisher was elected to the prestigious National Academy of Sciences in 2005.

² The authors note that the problems are probably not unique to neuroimaging papers in this particular subfield, however.

References

Eisenberger NI, Lieberman MD, Williams KD. (2003). Does rejection hurt? An FMRI study of social exclusion. Science 302:290-2.

Singer T, Seymour B, O'Doherty J, Kaube H, Dolan RJ, Frith CD. (2004). Empathy for pain involves the affective but not sensory components of pain. Science 303:1157-62.

Singer T, Seymour B, O'doherty JP, Stephan KE, Dolan RJ, Frith CD. (2006) Empathic neural responses are modulated by the perceived fairness of others. Nature 439:466-9.

Edward Vul, Christine Harris, Piotr Winkielman, & Harold Pashler (2009). Voodoo Correlations in Social Neuroscience. Perspectives on Psychological Science, in press. PDF

Vul E, Kanwisher N. (in press). Begging the question: The non-independence error in fMRI data analysis. To appear in Hanson, S. & Bunzl, M (Eds.), Foundations and Philosophy for Neuroimaging. PDF

Scan Scandal Hits Social Neuroscience

Mind Hacks uncovers a pre-print (PDF) by Vul, Harris, Winkielman, and Pashler entitled "Voodoo Correlations in Social Neuroscience". It's a "bombshell of a paper" that questions the implausibly high correlations observed in some fMRI studies in the field of Social Neuroscience. Vul et al. surveyed the authors of 54 papers to determine the analytic methods used. All but three of the authors responded to the survey, and 54% admitted to using faulty methods to obtain their results:

More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases.

A few of The Neurocritic's targets were on the hit list, so stay tuned.... there's more to come in 2009.

Deal, No Deal, or Dots?

OR: Is Perceptual Decision Making in Primate LIP Equivalent to Financial Decision Making Under Risk?

In the universally familiar game show Deal or No Deal, contestants choose from among 26 briefcases held by 26 models. Each of these briefcases contains a different amount of money ranging from $0.01 to $1,000,000. The contestant begins by choosing one briefcase, then starts selecting other cases to open, hoping to reveal small cash amounts because this will improve the odds of winning the $1 million. After a predetermined number of cases are opened, 'the Banker' tries to tempt the player to exchange her case for an amount of instant cash. The player must either stick with her original briefcase choice ('No Deal'), or make a 'Deal' with the Banker to accept his cash offer in exchange for whatever dollar amount is in the chosen case.

The show is a terrific example of financial decision making under risk. Nobel prize recipient Daniel Kahneman and his colleague Amos Tversky developed the idea of prospect theory to explain how people decide between alternatives that involve risk, when the outcome is uncertain but the probabilities are known (or estimated). Their highly influential paper (Kahneman & Tversky, 1979) framed these ideas, which provided an alternative to the expected utility hypothesis:



What does any of this have to do with dots??



The University of Rochester issued an egregiously erroneous press release to accompany the publication of a new paper in Neuron (Beck et al., 2008):

Our Unconscious Brain Makes the Best Decisions Possible

New Research Shows the Human Brain Computes Extremely Well—Given What it Knows

Researchers at the University of Rochester have shown that the human brain—once thought to be a seriously flawed decision maker—is actually hard-wired to allow us to make the best decisions possible with the information we are given.

Let's see, the study was done in monkeys (not humans), and the results said absolutely nothing about proving the brain is "hard-wired" to make the best decisions possible. The paper used computational methods to analyze the spike trains of neurons in the lateral intraparietal (LIP) area of monkeys who were trained to make motion discriminations. The original data were taken from the paper of Anne Churchland et al. (2008). One of the experimental tasks is illustrated below.


Figure 1A (Beck et al., 2008). Binary decision making. The subject must decide whether the dots are moving to the right or to the left. Only a fraction of the dots are moving to the right or the left coherently (black arrows). The other dots move in random directions. The animal indicates its response by moving its eyes in the perceived direction (green arrow).

Another variant of the task involved four choices instead of two (Churchland et al., 2008). In the Neuron paper, Beck et al. described a neural network model of decision making in these tasks. Although the motion direction task has been extensively studied in both animals and humans, the reported model is clearly based on recordings of LIP neurons in rhesus monkeys.

Back to paragraph #2 of the press release:

Neuroscientists Daniel Kahneman and Amos Tversky received a 2002 Nobel Prize for their 1979 research that argued humans rarely make rational decisions. Since then, this has become conventional wisdom among cognition researchers.

Kahneman and Tversky are/were (respectively) psychologists, not neuroscientists, and Tversky did not receive the Nobel prize. Perceptual discrimination of motion direction is not the same thing as financial decision making under risk, with its cognitive and affective elements. Although the monkeys were rewarded for correct decisions, reward functions were not a part of the network model. The authors summarized the significance of their work as follows:

First, we show that for Poisson-like distributions, optimal evidence accumulation can be performed through simple integration of neural activities, while optimal response selection can be implemented through attractor dynamics. Second, we show (again for Poisson-like distributions of neural activity) that neurons encode the posterior probability distribution over the variables of interest at all times. This latter contribution has far-reaching implications, since it suggests that neurons implicated in simple perceptual decisions represent quantities that are directly relevant to inference, confidence, and belief.

However, they didn't directly extrapolate their results to behavioral economics, and they didn't cite Kahneman and Tversky. Neverthess, the press release by the Senior Science Press Officer continues:

Contrary to Kahnneman and Tversky's research, Alex Pouget, associate professor of brain and cognitive sciences at the University of Rochester, has shown that people do indeed make optimal decisions—but only when their unconscious brain makes the choice.

At the risk of sounding pedantic, people did not make the decisions (monkeys did), and there was nary a mention of conscious vs. unconscious processing in the paper.

"A lot of the early work in this field was on conscious decision making, but most of the decisions you make aren't based on conscious reasoning," says Pouget. "You don't consciously decide to stop at a red light or steer around an obstacle in the road. Once we started looking at the decisions our brains make without our knowledge, we found that they almost always reach the right decision, given the information they had to work with."

Pouget says that Kahneman's approach was to tell a subject that there was a certain percent chance that one of two choices in a test was "right." This meant a person had to consciously compute the percentages to get a right answer—something few people could do accurately.

. . .

"We've been developing and strengthening this hypothesis for years—how the brain represents probability distributions," says Pouget. "We knew the results of this kind of test fit perfectly with our ideas, but we had to devise a way to see the neurons in action. We wanted to see if, in fact, humans are really good decision makers after all, just not quite so good at doing it consciously. Kahneman explicitly told his subjects what the chances were, but we let people's unconscious mind work it out. It's weird, but people rarely make optimal decisions when they are told the percentages up front."

I don't know if there would be any differences in the results if the monkeys were told the percentages up front... but you can watch Professor Kahneman discuss Decision Making and Rationality in Deal or No Deal Decisions, now showing on Channel N.

References

J BECK, W MA, R KIANI, T HANKS, A CHURCHLAND, J ROITMAN, M SHADLEN, P LATHAM, A POUGET (2008). Probabilistic Population Codes for Bayesian Decision Making Neuron, 60 (6), 1142-1152 DOI: 10.1016/j.neuron.2008.09.021

Churchland AK, Kiani R, Shadlen MN. (2008). Decision-making with multiple alternatives. Nat Neurosci. 11:693-702.

Kahneman D, Tversky A. (1979). Prospect theory: An analysis of decision under risk. Econometrica 47: 263-291.

Fainting In The Name Of

OR: The Glasgow Coma Scale-Revised: The Texting Sign.


Watch Killing in the Name, live at the Reading Festival 2008.



Rage Against the Machine Syncope

First we had dangerous sandwiches. Now we have dangerous concerts, as described in an article in the special Christmas edition of BMJ by Mike Sinclair and colleagues (Sinclair et al., 2008). They examined the utility of texting ability as a sign of return to consciousness after fainting or panic attack at large outdoor music festivals in the UK:

Three years ago we noticed that most of the patients with faint [syncope] or panic attack were teenagers and as soon as they could they used their mobile phones to send an SMS (short message service) text message to their friends...

The ability to text, whether or not it actually makes sense, requires a Glasgow coma scale score of 15 (fully conscious), an adequately functioning "executive system" in the frontal lobes, and a high degree of manual dexterity and psychomotor coordination. It also shows a degree of common sense not always evident in teenagers.

Two years ago we decided to use this texting sign as an indication that patients had recovered from their faint or panic attack and were orientated and coordinated enough to be discharged back to the festival. At times of massive influx to the medical tent, when up to two patients a minute are triaged, this system seems to work well.

The sets by Bloc Party and Rage Against the Machine were particularly busy times. The Festival Medical Services pit crew was able to treat

142 patients in less than 60 minutes during the performance by Bloc Party and 130 patients over 90 minutes during the performance by Rage Against the Machine. The texting sign needs further investigation to determine whether it is a valid criterion for recovery after faint or panic attack at festivals as well as in busy accident and emergency departments.

And now you do what they told ya (11 times)...

For another music-related article from the same issue of BMJ, see Between a rock and a hard bass in Mind Hacks.

Reference

M. Sinclair, D. W Pigott, K. N Carpenter (2008). Texting shows recovery after faint. BMJ, 337. DOI: 10.1136/bmj.a2723.

Dangerous Sandwiches

The "Enormous Omelet Sandwich" - 730 calories and 47 grams of fat



The "Monster Thickburger" - 1,420 calories and 107 grams of fat



The Ultimate Bacon Sandwich - 22 slices (1.25 lbs)



The 30,000 Calorie Sandwich (self-explanatory)


And now from The Lancet:

Dangerous Sandwiches

The unusual case of a woman who regularly fainted while eating sandwiches or fizzy drinks is explored in a Case Report in this week’s edition of The Lancet...

The 25-year-old woman was seen at the hospital in January this year. She presented with episodes, typically lasting 10 second or less, of feeling suddenly and alarmingly light headed, and nauseous. She had collapsed on more than one occasion, but had no movements typical of epilepsy. Sometimes she would have several episodes a week. The problem first began when she was 15 and remained unexplained despite hospital admissions between 2001 and 2007. A full battery of blood and other tests had, more than once, revealed everything to be normal. However, an electrocardiogram (ECG) test had shown a pause of 2.5 seconds. She then had external-loop ECG tests, in which she was asked to press a button to record 1-2 minutes of the ECG each time she felt faint. At times of light-headedness, she was found to have complete atrioventricular block (a slowing of intracardiac conduction), with beat-to-beat pauses lasting up to 2.5 seconds.

This was due to a rare case of swallow syncope (see also Armstrong et al., 1985 and Siegel, 2007)...

...a transient alteration or loss of consciousness during swallowing, and is usually intermittent. It may be caused by altered feedback in vagal reflexes—in which the afferent pathway, from the oesophagus, terminates in the nucleus tractus solitarius, and the efferent pathway runs from the medulla to the heart—or by vagal hypersensitivity.

After the patient was fitted with a pacemaker, her fainting episodes ceased and

When last seen, in June, 2008, she could eat sandwiches with impunity.

Let's hope she didn't choose the "Monster Thickberger"...

Reference

Christopher John Boos, Una Martin, Russell C Cherry, Howard J Marshall (2008). Dangerous sandwiches. The Lancet 372:2164.


I am making the most dangerous sandwich possible......with science!

Snotty or Nice?

Bhutta MF, Maxwell H. Sneezing induced by sexual ideation or orgasm: an under-reported phenomenon. J R Soc Med. 2008 Dec;101(12):587-91.

We describe a hitherto under-recognized curious response in some individuals: of sneezing in response either to sexual ideation or in response to orgasm. Our review suggests that it may be much more common than expected. We surmise that an indiscrete stimulation of the parasympathetic nervous system may be an underlying mechanism to explain this and other reported unusual triggers of sneezing.

First, we had the ACHOO (autosomal dominant compelling helio-ophthalmic outburst) syndrome, commonly known as photic sneezing:

The probable cause is a congenital malfunction in nerve signals in the trigeminal nerve nuclei. The fifth cranial nerve, called the trigeminal nerve, is apparently responsible for sneezes. Research suggests that some people have an association between this nerve and the nerve that transmits visual impulses to the brain. Overstimulation of the optic nerve triggers the trigeminal nerve, and this causes the photic sneeze reflex.

Now we have sneezing in response to orgasm or even thinking about sex. Here's a poor bloke writing in to The Times Online:

I’m a middle-aged man, and when I climax I have sneezing fits, which spoil the mood. Can I prevent this?

SUZI GODSON answers:

Impotence? Premature ejaculation? Inorgasmia? Sneezing? In the greater scheme of things, keeping a box of tissues by the bed is, I would suggest, a manageable sexual burden. After all, sneezing is not an entirely unpleasant experience. Indeed, it is often compared to orgasm because it is messy, takes about a second, requires copious Kleenex and strangers bless you afterwards.

If it makes you feel better, you are suffering from a serious-sounding but harmless medical condition. The propensity to sneeze at orgasm is closely related to the Achoo syndrome, commonly known as photic sneezing.

This is a genetically inherited neurological phenomenon, which occurs when exposure to bright light triggers pupil constriction in the eye. Pupil constriction is a function that is controlled by the brain stem, the multi-tasking lower part of the brain. The brain stem connects the spinal cord to the brain and plays a vital role in a smorgasbord of neurological functions that include breathing, digestion, heart rate, blood pressure and arousal. Because the neural pathways for these functions run very close to each other, it is thought that light-sensitive sneezing, sneezing after a meal or post-coital sneezing occur when the pathways get muddled up with the pathway that registers irritation in the nose.

Looks like she scooped Bhutta and Maxwell (2008) in her sex advice column of July 12, 2008...


See also violet blue ® on the subject, as well as Sneezing uncontrollably after sex may be more common than realised, The sexy secret that could lie in a sneeze, and (my favorite) Snot so innocent.