In 1936, a physicist submitted a paper to a physics journal and was taken aback when the journal’s editor returned to him and his coauthor a set of reviewer critiques of the work. Insulted, the physicist withdrew his paper from The Physical Review. Albert Einstein and his coauthor submitted the paper elsewhere where it was published, and later the paper’s conclusions were revealed to be wrong. Most of Einstein’s work, including his groundbreaking papers, may not have been peer reviewed. But in this case, the peer review process did spare The Physical Review the ignominy of publishing an incorrect result by the famous physicist.
Reaction to the recent investigation by Science journalist John Bohannon has included the allegation by Michael Eisen that peer review is a joke. Last week the magazine published a report of Bohannon’s submission of a flawed paper to numerous open access (OA) journals, and Eisen immediately denounced the study. But I doubt the American Society of Microbiology, Lippincott Williams and Wilkins, Nature Publishing Group, the Public Library of Science, or Springer Verlag feel that peer review is a joke. Their open access journals rejected Bohannon’s bogus paper as the result of their review processes. Meanwhile, OA journals operated by two of their competitors, Sage Publications and Elsevier, fell for the hoax and accepted the paper. Doubtless, those two otherwise reputable publishers are critically evaluating their journals’ review processes now.
How valuable is peer review to a publication’s reputation? If a traditional (i.e., subscription-based) journal had accepted Bohannon’s bogus paper, would that journal still be regarded as valuable by the research community? Would that journal perhaps be more susceptible to cancellation by librarians under fiscal pressures? Eisen has noted that Science itself, after peer review, published a paper on arsenic-based life which was refuted. According to a Science spokesperson, however, the incident prompted the magazine to refine its review process so that reviewers can see one another’s comments.
One incorrect result slipping through peer review does not indict the entire practice. Indeed, many researchers value peer review as a quality marker that challenges them to refine and improve their research. On the other hand, when 157 OA journals accept a deeply flawed paper, then something is awry.
Researcher Jeff Shrager has suggested not only that peer review is valuable but that there may be a sweet spot between how high and how low the bar is set–if any bar is set at all. From participating in numerous editor meetings in a variety of disciplines, I know that journal editors constantly struggle with the quality of reviews and how high to set the bar. But for the sake of one’s reputation, there is no question that peer review is worth the endeavor.