Monday, April 21, 2008

Double-blind peer review

Double-blind review is a method of peer review used by some scholarly journals. Most journals use single-blind review, in which the identity of the reviewers is not (necessarily) revealed to the authors of manuscripts. In double-blind reviewing, the author(s) of the manuscript are not revealed to the reviewers. This requires editors to cut out the author name and perhaps the acknowledgements section. This is presumably done to improve the objectivity of the peer review process, but I have my doubts.

The reasons for keeping the names of reviewers hidden from authors (single-blind reviews) are pretty clear. I am more willing to be critical of the manuscripts of friends and colleagues, or of prominent scholars, if I know that my identity will not be revealed to them. There are criticisms of blind reviewing: reviewers can be petty and nasty and hiding behind anonymity does not help the overall process. But single-blind reviewing is a standard procedure in archaeology. Some journals today give reviewers the choice of remaining anonymous or revealing their name. I usually sign my reviews with my name. But sometimes (maybe 20% of the time) I don’t want to do this. I feel it is my duty to be rigorous and critical in my reviews, and sometimes friends and colleagues submit shoddy papers that I must criticize. Keeping anonymity can help avoid unpleasant personal repercussions, while maintaining professional standards.

I am less enthusiastic about trying to hide the names of authors from reviewers. There are both logistical and professional reasons for this. On the practical side, I have never reviewed a “blind” manuscript without knowing exactly who wrote it. I typically know who is doing what in my field, and even if that isn’t clear, there are usually telltale signs in the text that the journal editors overlook. Both of these factors were present in a manuscript I recently reviewed using the double-blind method.

But there is a larger professional and scientific reason to question double-blind reviewing. Manuscripts do not exist in a professional vacuum. Published papers are almost always part of a body of work (by an author or authors, or by a project or a group), and part of the evaluation of a paper focuses on its role within that body of work. If one really doesn’t know the identity of the author, then it is difficult or impossible to place the paper in its context.

Suppose I am reviewing a boring and not particularly rigorous or enlightening paper on obsidian production at a Mesoamerican site. If that paper were authored by an important obsidian analyst (say, Ken Hirth or John Clark), my reaction as a reviewer would be to tell the author to go back and improve the paper. There is no reason to accept shoddy work by these people (in point of fact, I don’t know of any shoddy papers by John or Ken, whose work is very good). On the other hand if the paper is by someone of questionable ability with a poor record of publication who happened to excavate an important obsidian assemblage years ago and is on the point of retiring, then perhaps a shoddy manuscript should be published. It would be much better for the discipline to have a low-quality publication from such a person than to have no publication at all.

There is a nice new British report on peer review: British Academy (2007) Peer Review: The Challenges for the Humanities and Social Sciences. The British Academy, London.


V. Page said...

I am absolutely for the double-blind peer review because I think it makes the system much more objective (actually I thought it was the norm!):

1. It is petty and pathetic when colleagues you have fallen out with or dislike you for their own reasons take their revenge when reviewing your paper by being nasty and nihilistic, hiding behind anonymity;

2. On some other occasions, when the anonymous reviewer reads "an important name" s/he is either extremely critical OR extremely favourable towards the writer. However, the scientific reviewer must be objective above all.

I have been a victim and a witness of both situations and I get furious by these approaches. It is high time the omnipotence of "anonymous reviewers" came to an end by giving the writer a fair chance as well.

PS. I have been a reviewer in a double-peer review myself and I felt a feeling of absolute objectivity, within the limits of my flawed human nature, without any "names" or "guilt" or "nastiness" getting in the way between me and the manuscript.

Highly recommended!

Michael E. Smith said...

These are good points, and I find it difficult to take an absolute stand on the issue. In my thinking, the negatives of double-blind reviewing generally outweigh the positives. I do know that anthropology and archaeology journals that have tried double-blind reviewing have abandoned it pretty quickly. There are probably major differences among disciplines that lead toward or away from the double-blind process. Some relevant factors might be the size of the community of scholars, the degree of tight competition among researchers for credit, etc.

Michael E. Smith said...

My views on blind peer review were influenced by: Michael T. Ghiselin (1989) Intellectual Compromise: The Bottom Line. New York: Paragon House. See p. 64. I tried to find this citation when I wrote the entry, but only now came on it in an old folder of papers.