Monday, March 21, 2016

Empty citations

In my paper on arguments in archaeology (Smith, Michael E.,  2015,  How can Archaeologists Make Better Arguments? The SAA Archaeological Record 15(4):18-23) I mention the problem of empty citations. A comment this weekend on Twitter got me to look again at the basic reference on empty citations:

Harzing, Anne-Wil
2002 Are our referencing errors undermining our scholarship and credibility? The case of expatriate failure rates. Journal of Organizational Behavior 23:127-148.

Harzing lists "12 guidelines for good academic referencing." I listed these in an earlier post in 2008. They bear repeating, so here they are:
  1. Reproduce the correct reference. This means get the details of the citation right.
  2. Refer to the correct publication. For example, do not cite Binford (1972) for something that was in fact said in Binford (1965).
  3. Do not use “empty” references. “Empty references do not contain any original data for the phenomenon under investigation, but strictly refer to other studies to substantiate their claim.”
  4. Use reliable sources.
  5. Use generalizable sources for generalized statements. In other words, don’t cite a single example as if it provided support for a more general phenomenon, and don’t cite a study of X artifacts at Y site as if it pertained to many more kinds of artifacts at many sites.
  6. Do not misrepresent the content of the reference.
  7. Make clear which statement references support. Don’t include a number of claims in one sentence, and then append a bunch of references.
  8. Do not copy someone else’s references.
  9. Do not cite out-of-date references. The point here is to avoid data and interpretations that have been discarded or superseded. Many archaeological data reports, of course, NEVER go out of date.
  10. Do not be impressed by top academic journals.
  11. Do not try to reconcile conflicting evidence.
  12.  Actively search for counter-evidence.
Harzing found that the greatest problems were caused by violations of guidelines 3, 4, 6, and 9. It is my subjective impression that in archaeology, guidelines 3, 4, 11 and 12 are particularly subject to violation. I will refrain from naming names here in order to avoid antagonizing people.

For a parallel study, published in New Scientist, see: “Scientists exposed as sloppy reporters”

I though I would give an example here. I'll leave out the names, to protect the guilty.

Scholar A publishes a book chapter (in 1994) that analyzing some information. This person then makes a claim about what ancient people in Mesoamerica were trying to do in certain circumstances. This claim has ABSOLUTELY NO empirical basis; it is a speculation based on the worst of flimsy evidence. It is based more on Scholar A's preconceived ideas than on evidence. But instead of framing this claim in hypothetical terms, it is presented as a simple statement of fact. I happen to know the data and research tradition well, and you are going to have to accept my claim that I am not misrepresenting the facts. Maybe I will reveal the original paper, but leave the empty citations anonymous.

Then archaeologists, who share Scholar A's preconceived ideas, start citing this chapter as supporting the idea about what ancient Mesoamericans were trying to do. This appears in at least 3 articles in Latin American Antiquity (between 2004 and 2011), it appears in four or five chapters of an edited volume published by a reputable press in 2005, and it appears again in an edited volume published in 2016, citing not only the original paper, but some of the earlier works citing the original paper. These are all empty citations:
Empty references are references that do not contain any original evidence for the phenomenon under investigation, but strictly refer to other studies to substantiate their claim. Other authors subsequently use these empty references to substantiate their claims rather than going back to cite the original source (Harzing 2002:130).
If you haven't read my paper on arguments, please take a look. And also take a look at Harzing. It is ridiculous that this practice is common in archaeology and Mesoamerican studies.

Harzing, Anne-Wil
    2002    Are Our Referencing Errors Undermining our Scholarship and Credibility? The Case of Expatriate Failure Rates. Journal of Organizational Behavior 23:127-148.

No comments: