SUBSCRIBE NOW!

Vol. 20 No. 4 — April 2003

But two scholarly societies are now asking if peer review is in fact such a good thing.As the discussion about finding alternative ways of distributing research has intensified over the past few years, it's not surprising that peer review itself would eventually come under scrutiny. And two U.K. groups, the Cochrane Collaboration and the Royal Society, are suggesting that there's a need to examine the value of peer review and to determine whether its methodology should be systematically revised.

The motivation for this inquiry is concern that the public is losing faith in peer review's ability to create a gold standard for evaluating research quality. Last year in the U.S., there were several instances of significant peer-review failures. The most startling was revealed last October when the work of Bell Laboratories' Jan Hendrick Schon came under scrutiny. Schon published 25 papers over the past 3 years. Of those, 16 have been declared to be false. This finding caused the prestigious journal Science to withdraw eight of his papers.

Cochrane Review

In January, Cochrane published "Editorial Peer-Review for Improving the Quality of Reports of Biomedical Studies." According to the organization, this study was conducted because "the knowledgebase of peer review has traditionally lagged far behind its acceptability and use as a quality-assessment tool." Cochrane claims that assumptions about the use of peer review for objective decision making have "rarely been tested." In addition, it says that the need to "prioritize information sources is crucial since there are over 20,000 biomedical journals published globally."

Cochrane, founded in 1992 in Oxford, U.K., is an international organization that "aims to help people make well-informed decisions about healthcare by preparing, maintaining, and ensuring the accessibility of systematic reviews of the effects of healthcare interventions."

The report began with an initial look at 135 studies on the effects of peer review in biomedicine. However, the Cochrane reviewers found that only 21 offered credible data. They concluded that "given the amount of resources invested in peer review and the common assumption that it improves such outcomes, the lack of research documenting improvements is striking." In addition, many of the studies show that the "well-researched practice of concealing the identities of peer-reviewers or authors, while laborious and expensive, appears to have little effect on the outcome of the quality-assessment process."

So it's not surprising that the Cochrane reviewers argue, "A well-funded and coordinated effort involving several sectors of the scientific community is required to improve our understanding of the effects of peer review." They claim that such a study "would best be coordinated by a scientific council ... with the specific aim of scientifically assessing the quality of peer viewing." Until a study is undertaken, the reviewers caution that "we are not confident in the present system" of peer review.

Because the peer-review process is so deeply entrenched in the research system, this fascinating study would be a formidable undertaking. To many, defending the process is like upholding the American virtues of motherhood, baseball, and apple pie. But since Schon was able to slip past Science's venerated quality-control mechanisms more than a half-dozen times, it's appropriate to ask if traditional peer-review practices can still be applied in an environment that offers so many publishing outlets and places so many demands on the reviewers' time.

Royal Society Working Group

Shortly after the release of the Cochrane review, the Royal Society, the U.K.'s leading research institution, announced it was going to examine best practices in peer review. It has set up a 10-member working group to oversee this process. Heading up the inquiry is Patrick Bateson, provost of King's College as well as biological secretary and vice president of the Royal Society. He believes that "peer review is an imperfect process."

In a February interview in The Guardian, Bateson argued: "Scientists are under enormous pressure these days, and many are reluctant to give the time to [peer review]. Sometimes what happens is that the paper gets passed to a graduate student who then delivers a damning critique." He stated further: "We are all aware that some referees'reports are not worth the paper they are written on. It's also hard for a journal editor when reports come back that are contradictory, and it's often down to a question of a value judgment whether something is published or not."

The Royal Society Working Group has established an aggressive time frame for generating two reports by September. One of the committee's charges is to determine best practices for peer review and consider alternatives such as naming the referees. This system might make referees more responsive and responsible (and perhaps more polite).

A second report aims to help the public interpret scientific results. This could be a particularly compelling document if the working group concludes that the existing peer-review system is in fact fundamentally flawed. It would place the Royal Society in the awkward position of trying to calm public fears about the trustworthiness of scientific research while calling for essential changes in the very framework that underpins it.

If the Royal Society does call for new best practices for peer review, it will be interesting to see how or if the research community embraces them.