The journal Proceedings of the National Academy of Sciences (PNAS) publishes papers in all the sciences, including archaeology. I have long wondered whether the peer review process of PNAS us up to snuff. It seems that a good number of papers published in the journal fail to cite relevant sources on prior research. PNAS members can publish in the journal, and I'm not sure if their papers are peer reviewed or not. Non-PNAS members evidently can have their papers reviewed by a single PNAS member, who may be a good friend. Hardly a rigorous review process, but then perhaps that is not the goal.
I just read John Hawks's blog entry on this (from January, 2011), "Membership has its privileges." He takes a paper by Erik Trinkaus on Neanderthals in the PNAS to task for ignoring prior research very similar to Trinkaus's. Interestingly, Hawks publishes his own email correspondence with Trinkaus about the issue. It is worth reading.
A similar case occurred more recently, when a paper in PNAS on agricultural intensification failed to cite the literature on the topic. Sam Bowles shows that agriculture is calorically less efficient that hunting, something I learned as an undergrad more than thirty years ago. I complained to some people in the NAS about this, and had a short email exchange with Bowles. There is quite a large literature on this in anthropology, archaeology, and geography, much of it in the 1970s and 1980s (Boserup and her implications, pretty basic stuff in archaeology), but Bowles presented his findings as if he were the first to figure out the energetic costs of intensification:
Bowles, Samuel
2011 Cultivation of cereals by the first farmers was not more productive than foraging. Proceedings of the National Academy of Sciences 108:4760-4765.
I was tempted to fire off a letter to the journal, but desisted for two reasons. First, intensification is not my own specialty, and I thought there might be complexities and literature I am not aware of. Second, my previous letter to the editor was not very satisfying. In that case, I criticized a PNAS paper by economic historians for sloppy terminology and concepts:
Basu, Sudipta, John Dickhaut, Kristy Towry, and Gregory Waymire
2009 Recordkeeping Alters Economic History by Promoting Reciprocity. Proceedings of the National Academy of Sciences 106:1009-1014.
The authors claimed that the results of a modern economic game explained past developments in economic history. I pointed out that several intermediate inferential steps had been left out; one cannot read ancient history from the results of a modern game. This is my letter:
2009 Modern Behavioral Experiments are not Economic History (response to S. Basu, J. Dickhaut, K. Towry, & G. Waymire, “Recordkeeping Alters Economic History by Promoting Reciprocity,” PNAS 106: 1009-1014). Proceeding of the National Academy of Sciences 106 (16), p. E39.
Like John Hawks, I am dismayed that prominent scholars are permitted to publish papers in a prestigious journal like PNAS without citing the relevant literature. If I tried to do that in a basic archaeology journal, I'd almost certainly be shot down by the reviewers.
3 comments:
Among the more conspicuous examples in archaeology of the odd peer review process at PNAS for academy members are a couple of papers by Flannery et al. (2005) and Stoltman et al. (2005), attacking a paper published in Science by Blomster et al. (2005). The issue was the validity of a stunningly comprehensive INAA study showing that ceramics from the Olmec center at San Lorenzo were exported to sites in surrounding regions, but no pots moved from those regions to San Lorenzo.
One can imagine several hypotheses to account for the fact that the critiques did not appear in the journal in which their target was published – is there some truth after all to the old canard that PNAS stands for “Publications Not Acceptable in Science?
Portions of the resulting exchange can be found on Hector Neff’s website: http://www.csulb.edu/~hneff/, including the original paper in Science and two replies, from Neff and colleagues, which do a superb demolition job on Flannery and Stoltman attacks.
The reply to Flannery and Stoltman appeared in Latin American Antiquity in 2006, not PNAS, which may raise additional issues about PNAS’s review policies: does PNAS refuse to publish responses critical of papers published in PNAS by academy members? The second Latin American Antiquity paper by Neff et al. contains an illuminating discussion of PNAS review policies.
Two thoughts about this exchange. As Neff et al. effectively showed, Flannery and Stoltman critiques are full of flaws. So the apparently cursory review they initially received at PNAS ended up working in against them and favor of the original study. On the other hand, the fact that those flaws, especially the statistical ones, made it into print in PNAS makes this exchange a wonderful resource on which to draw when teaching compositional data analysis. If you ever need a great example of how not to use linear discriminant functions, PNAS provides one.
Here is the sequence:
Blomster, Jeffrey P., Hector Neff, and Michael D. Glascock 2005 Olmec Pottery Production and Export in Ancient Mexico Determined through Elemental Analysis. Science 307:1068-1072.
Flannery, Kent V., Andrew K. Balkansky, Gary M. Feinman, David C. Grove, Joyce
Marcus, Elsa M. Redmond, Robert G. Reynolds, Robert J. Sharer, Charles S. Spencer,
and Jason Yaeger 2005 Implications of New Petrographic Analysis for the Olmec “Mother Culture” Model. Proceedings of the National Academy of Sciences 102:11219-11223.
Stoltman, James B., Joyce Marcus, Kent V. Flannery, James H. Burton, and Robert G.
Moyle 2005 Petrographic Evidence Shows that Pottery Exchange Between the Olmec and
Their Neighbors Was Two Way. Proceedings of the National Academy of
Sciences 102:11213-11218.
Hector Neff, Jeffrey Blomster, Michael D. Glascock, Ronald L. Bishop, M. James Blackman, Michael D. Coe, George L. Cowgill, Richard A. Diehl, Stephen Houston, Arthur A. Joyce, Carl P. Lipo, Barbara L. Stark, Marcus Winter 2006 Methodological Issues in the Provenance Investigation of Early Formative Mesoamerican Ceramics. Latin American Antiquity 17(1):54-76.
Robert J. Sharer, Andrew K. Balkansky, James H. Burton, Gary M. Feinman, Kent V. Flannery, David C. Grove, Joyce Marcus, Robert G. Moyle, T. Douglas Price, Elsa M. Redmond, Robert G. Reynolds, Prudence M. Rice, Charles S. Spencer, James B. Stoltman, Jason Yaeger 2006 On the Logic of Archaeological Inference: Early Formative Pottery and the Evolution of Mesoamerican Societies. Latin American Antiquity 17(1): 90-103.
Hector Neff, Jeffrey Blomster, Michael D. Glascock, Ronald L. Bishop, M. James Blackman, Michael D. Coe, George L. Cowgill, Ann Cyphers, Richard A. Diehl, Stephen Houston, Arthur A. Joyce, Carl P. Lipo and Marcus Winter 2006 Smokescreens in the Provenance Investigation of Early Formative Mesoamerican Ceramics Latin American Antiquity 17(1): 104-118
This is an excellent question that I keep thinking about. It seems challenging, no? Isn't it hard sometimes to cite all the relevant literature? Even key studies? Also, it seems so much knowledge in archaeology, and other fields in the social sciences, reappears in cycles. Also, at what point do particular theories become more common knowledge. Let's take the intensification is less efficient than foraging. i do not believe Boserup ever said this, though it is kinda an implication of her work (actually what she said was that under increasing population growth, intensification is MORE efficient in terms of per capita labor investment given diminishing marginal utility...that is why intensification occurs, for her....). Anyway, Richard Lee says it in many ways, as does Sahlins, Johnson, etc. But even these folks only strategically cite one another. At what point can one tell that all appropriate stuff has been cited? For example, if I was to say that populations eventually will reach a carrying capacity, who would I cite? If I just cited Malthus, would that end it? Or would I have to go back before him? If I was to say, for example, "Empires involve the dominance of one polity by another," who would I cite here? I understand a problem in citing methodological advances, however. But, what should one do? I think this is such an important question that you raise, especially for younger scholars. I have always heard, one can either write or read the literature. Or maybe you have already provided the answer. As authors we all make mistakes and omissions. It is part of the responsibility of reviewers and editors to point these things out to us (?)...thanks for this.
@Anonymous #1 - I'm not inclined to discuss the Olmec debate here. Neither side is blameless (intellectually and professionally), and I don't have much to contribute. The debate played out in a very public context. But thanks for pointing out the link to the issue of reviewing at PNAS.
Post a Comment