Peer Review

The aim of Peer Review is to make sure that publications and grant submissions meet an acceptable standard as defined by a community of peers (The idea of trial by your peers goes back to Magna carta). The question is does it work?

Only 8% members of the Scientific Research Society agreed that 'peer
review works well as it is.' (Chubin and Hackett, 1990; p.192)

"A recent U.S. Supreme Court decision and an analysis of the peer review system
substantiate complaints about this fundamental aspect of scientific research."
(Horrobin, 2001)

"is a non-validated
charade whose processes generate results little better than does chance."
(Horrobin, 2001)

"Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein,
2000)

"Peer Review is central to the organization of modern science…why not apply
scientific [and engineering] methods to the peer review process" (Horrobin,
2001).

Currently most peer review is carried out anonymously. That is the reviewers are anonymous but the authors of a paper or grant proposal are known to the reviewer. Often when you submit a paper you are asked for a list of potential reviewers, some of whom will be used to review your paper. These two features of the review process often combine to make success or failure of a paper submission dependent on how good the submitting author is at "networking". Well known names with a large circle of connections select members of this group as their reviewers and so they are more frequently published and become more famous as an expert in that area and have more connections. This is a "rich get richer" scenario, which makes it very difficult for new researchers to break into a field. Especially a field which is fiercely competitive and in which the leading members have large egos.

Some of the open source journals are proposing alternatives:

1. Publish the reviews alongside the paper.
2. Name the reviewers so they are no longer anonymous to the authors.
3. Anonymise the papers so that the reviewers do not know who the authors are.
4. Publish the papers online before peer review and let the community review them before they take a fixed form.

The consequences of (1) are to moderate the language of reviews. Reviewers are likely to be more careful about what they say if they know these reviews are going to be made public. Combining (1) and (2) is even better as it forces reviewers to act objectively as their reputation amongst the community matters. Idea (3) is hard to achieve as reviewers will be able to work out who the authors are from a knowledge of their field and the literature cited (all scientists cite themselves more than others). The problem with (4) is how active will the community be? If there is a very active community in a small field this might work, but there is still the question of the status of the papers in the depository before they reach their final form when they are effectively not peer reviewed.

The quotes above were taken from an e-mail advertisement for a conference on improving peer review.

References

Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy; New York, State University of New York Press.

Horrobin, D., 2001, "Something Rotten at the Core of Science?" Trends in Pharmacological Sciences, Vol. 22, No. 2, February 2001. Also at http://www.whale.to/vaccine/sci.html and http://post.queensu.ca/~forsdyke/peerrev4.htm (both Web pages
were accessed on February 1, 2010)

Goodstein, D., 2000, "How Science Works", U.S. Federal Judiciary Reference Manual on Evidence, pp. 66-72 (referenced in Horrobin, 2000)