medical examiner

Quality Control
The case against peer review.
By Daniel Engber
Posted Tuesday, April 5, 2005, at 11:03 AM PT

In September 2001, the Journal of Reproductive Medicine weighed in on the healing power of God. A Columbia University research group reported that patients at a fertility clinic in Seoul were twice as likely to get pregnant when Christians prayed for them. Within a month, the study was in the New York Times science section and on Good Morning America, where the medical editor for ABC News called it "very well done" and opined that "getting pregnant involves a lot of biological, psychological, maybe even spiritual factors that we don't yet understand."

The prayer study has since fallen from grace. Scientists around the world wrote angry letters to the journal attacking the methodology, and the research-protections office of the Department of Health and Human Services looked into whether the subjects had properly given consent. Last year, the study's senior author removed his name from the paper, saying that he hadn't directly participated in the research. The real lead author will not discuss the work, and the third author—a parapsychologist, lawyer, and convicted con man—is now serving time in a federal prison (for an unrelated charge of fraud).

Why did this quackery get so far before being exposed? The prayer study seemed legitimate because it appeared in the pages of a "peer-reviewed" medical journal. That means the paper was vetted by an independent panel of experts in the field.

Peer review is the gold standard of modern science. For medical researchers and other scientists, it's the gateway to funding, publication, and career advancement. When they apply for government grants from the National Institutes of Health or the National Science Foundation, their proposals are reviewed by a panel of their colleagues. When they submit their completed work for publication, journals and university presses ask for the opinions of others in the field. And when they apply for jobs or tenure, scientists are judged largely on the basis of their peer-reviewed publications.

Scientists give peer review so much authority because they view it as a part of the grand tradition of scientific inquiry—an extension, even, of the formal experimental method. Peer evaluation is the endpoint of a cautious progression from theories and predictions to experiments and results. The system dates from the 1700s, when the Royal Society of London set up a "Committee on Papers" with the power to solicit expert opinions. It became the standard for scientific publication only after World War II, when the dramatic expansion of scientific research swamped journal editors and made them look to outsiders for help. Ever since, scientists have claimed that peer review filters out lousy papers, faulty experiments, and irrelevant findings. They say it improves the quality of an accepted paper by providing helpful comments for revision. And they can't imagine a better way to accomplish these goals.

So, what explains the Columbia prayer study? Journal editors will tell you that peer review is not designed to detect fraud—clever misinformation will sail right through no matter how scrupulous the reviews. But the prayer study wasn't a clever fraud. It was sprinkled with suspect elements, not the least of which was a set of results that violated known laws of science. The authors also used a needlessly convoluted experimental design; these and other red flags in the study have been cataloged on the Web by obstetrician and enthusiastic debunker Bruce Flamm. Even on its own terms, then, as a filter for lousy papers and bad experiments, peer review of the Columbia prayer study was a spectacular failure. Here's the problem: Despite its authority and influence over every aspect of the scientific community, no one has ever shown that peer review accomplishes anything at all.

Daniel Engber is a writer in New York City.

Article URL: http://slate.msn.com/id/2116244/