One benefit of the epistemology part of the class is that it has helped me organize my thoughts, and given me a better understanding of just what is wrong with much of the work published in archaeology today. In short, many archaeologists don't know how to make a solid argument. I've talked about this previously. They don't know how to put data together with theory to reach a rigorous conclusion about what likely happened in the past and why it happened. So, I plan to write an article about this problem. Right now I view the paper partly as instructions for students (here is how to make a good argument, and here are some pitfalls to avoid), and partly as a critique for the profession. One difficulty in writing this paper will be how to handle examples. Maybe I am getting soft in my old age, but I am not anxious to get a bunch of colleagues pissed off at me for featuring their work as negative examples. I seem to do a pretty good job of annoying people without deliberately poking a bunch of others with a stick. But would a journal editor accept an article about methodological and epistemological problems that did not include a lot of real examples?
Anyway, I think I'll use this blog to organize some of my thoughts for my paper on arguments. So let's start with a quote from economic historian Steve Haber:
- “the fundamental question of all serious fields of scholarly inquiry [is]: How would you know if you are wrong?”(Haber 1999:312)
Here is how Andrew Abbott (2004) discusses the problem of ensuring that you can be wrong:
- “it is surprising how many researchers—even graduate students in their dissertations—propose arguments that can’t be wrong. For example, research proposals of the form, ‘I am going to take a neo-institutionalist view of mental-hospital foundings’ or ‘This paper analyzes sexual assaults by combining a Goffmanian account of interaction and a semiotic approach to language’ are not interesting because they do not propose an idea that can be wrong. They boil down to classifying a phenomenon or, seen the other way around, simply illustrating a theory.” (p.216)
Abbott goes on to remark that,
- “Thinking without alternatives is a particular danger in ethnography and historical analysis, where the natural human desire to develop cohesive interpretations (and the need to present a cohesive interpretation at the end of the research) prompts us to notice only those aspects of reality that accord with our current ideas.” (p.216)
Does this sound familiar? As archaeologists we certainly want to write cohesive narratives, and this desire may lead us away from considering alternative ideas. One of the problems with using post-hoc interpretations, or what Lewis Binford called post-hoc accomodative arguments (see my prior discussion here), is that they can't be wrong. They are made up after the fact, after the research is done, and they are designed to fit the results you found. So almost by definition, such interpretations must be correct. They fit the data, and you haven't compared them to any other interpretation. Now there is nothing wrong with devising some interpretations after all the facts are in. But such interpretations should serve as input for further research that is designed to test them against new data. Otherwise they will remain a particularly weak argument.
So, how do you avoid doing research and constructing arguments that can't be wrong? Here are two suggestions:
(1) Devise multiple working hypotheses, and test them all. The one that best survives its battle with the empirical world, the one that is the last hypothesis still standing, is your best explanation. This concept of "multiple working hypotheses," usually with a mandatory citation of Chamberlin, comes up occasionally in archaeology. In fact, this method is part of the approach known as "strong inference." Want to hear more? Stay tuned. This will be one subsection of my paper, and I'll talk about it here before too long.
(2) Choose your theory wisely. If your theory is very abstract and philosophical, at a high epistemological level, then you can't test it against data, and you can't show that your interpretation is wrong. On the other hand, if your theory is of the middle range or low range (what I have called "empirical theory" - Smith 2011), then it CAN be tested against the empirical record, and it can be falsified.This is another well-established point in social science epistemology (see, for example, Ellen 2010, or see the discussion in my 2011 paper), although the post-processualists don't believe that there are different epistemological levels of theory (see footnote 4 to my 2011 paper on this).
For example, suppose I am working on regional ceramic exchange in Postclassic central Mexico. I could decide to test a middle-range theory, such as the proposition that "increasing involvement in the long-distance exchange networks of a commercialized economy will lead to higher levels of local and regional exchange at my site." I know from past work, and work in other areas, that the Late Postclassic period was a time of growing commercialization and increasing long-distance trade throughout Mesoamerica. I find more long-distance imports in my site through time. So I do some petrographic analysis and INAA of a sample of sherds from different time periods. I may find my hypothesis supported (that is, there are higher frequencies of regional imports through time), or I may find that I was wrong (frequences of local/regional imports declined when I expected them to increase). This kind of research, using middle-range theory, can lead to results that either confirm or deny my initial hypothesis. (or, they may lead to complex results that leave me scratching my head.....).
Now, consider an alternative use of theory. Suppose I choose to use a high-level abstract theory: post-structuralism. The growing influence of commercial exchange created social contradictions and tensions in local society. Political institutions at my site were instantiations of socially embedded practices. People negotiated their identity by using ceramics from different places of origin in arenas of display and tournaments of value. My analytical tests reveal the places of origin of regionally imported ceramic types, consistent with the notion of a fragmented and contested, yet socially embedded, economy. Now, can this interpretation be wrong? How could I convince a skeptic that it is correct and supported by data? High-level, abstract social theory simply cannot be tested, and it cannot be directly evaluated with data.
Anyway, sorry to go on at such lengths. I'm sure that clever social theory types will recognize my postcolonial scenario as a bunch of gobbledygook. But I think this issue of "how would you know if you are wrong?" lies at the heart of many problems of research and writing in archaeology today. Now, maybe some archaeologists don't see the need for rigorous testing of hypotheses or for designing arguments that can be proven wrong. Maybe a scientific epistemology is not valuable to these people. Maybe they don't want to be competitive for grants from the National Science Foundation. Fine. But for the rest of us, we should strive to make better arguments. If you haven't read Booth et al. (2008), that is probably the best place to start.
My paper about arguments will also cover good and bad uses of analogy, the argument template contained in Booth et al (2008), empty citations, natural experiments, strong inference, causal mechanisms, and perhaps even Monty Python's Argument Clinic (I've always wanted to cite a Monty Python sketch in a serious scholarly paper, and this may be my best chance yet).
2004 Methods of Discovery: Heuristics for the Social Sciences. Norton, New York.
Booth, Wayne C., Gergory G. Colomb, and Joseph M. Williams
2008 The Craft of Research. 3rd ed. University of Chicago Press, Chicago.
Ellen, Roy
2010 Theories in Anthropology and "Anthropological Theory". Journal of the Royal Anthropological Institute 16: 387-404.
Gerring, John
2007 Case Study Research: Principles and Practices. Cambridge University Press, New York.
Haber, Stephen
1999 Anything Goes: Mexico's "New" Cultural History. Hispanic American Historical Review 79: 309-330.
Luker, Kristin
2008 Salsa Dancing Into the Social Sciences: Research in an Age of Info-glut. Harvard University Press, Cambridge.
Popper, Karl R.
1934 The Logic of Scientific Discovery. Harper and Row, New York.
Ragin, Charles C. and Lisa M. Amoroso
2011 Constructing Social Research: The Unity and Diversity of Method. 2nd ed. Sage, Thousand Oaks, CA.
Smith, Michael E.2011 Empirical Urban Theory for Archaeologists. Journal of Archaeological Method and Theory 18: 167-192.
6 comments:
Great post, Mike. I think we need to be careful not to toss out the baby with the bathwater, however. Not all "high-level theory" is the same. Some of it is untestable, for sure. But certain other bodies of high-level theory do make predictions that are testable. Behavioral Ecology and Dual Inheritance Theory are the two that spring most readily to mind. I think we need to urge folk to be more selective in their choice of high-level theory rather than encouraging them to abandon it. Dropping high-level theory and focusing solely on middle-range theory would make archaeology even more irrelevant to mainstream science than it already is, I fear.
@Mark - You are absolutely right, and I shouldn't complain about high-level theory in general. I am concerned about the problems with high-level theory that lacks operationalization, theory that does not have clear lower-level derivations that can be tested. This obviously excludes HBE and dual-inheritance theory,both of which are clearly linked with operational methods and lower-level concepts. My larger point focuses on testing, and these bodies of theory clearly are based on testable (and tested) propositions. But approaches like practice theory and some kinds of materiality theory, postcolonial theory lack the lower-level operationalization that permits testing.
Is it that such theories are not operationalizable, or that their proponents don't appreciate the need to operationalize them and/or don't know how to operationalize them? A few years ago Ted Slingerland and I made the case that the hypothesis-testing approach could and should be extended to most humanistic theories (intro chapter in Slingerland and Collard [2011] "Creating Consilience: Reconciling Science and the Humanities"), and I still think there's merit to that argument. I suspect that much high-level social theory in archaeology could be operationalized if people were so inclined, and that it's the lack of inclination that's the biggest problem. That and the lack of proper training in research design.
I am agnostic on whether the problem is the nature of the theories, or the inclinations of the people who use them. Practice theory has been an infatuation of interpretive archaeologists for a long time, and I would think that if it could be operationalized effectively, we would have seen that by now.
My view is pretty standard among social science practitioners, some of whom I cite in my urban theory paper. A high-level theory explains a lot at a very abstract or general level that does not relate closely to the empirical world. A lower-level theory explains far less, but with much greater fidelity to data. The paper by Roy Ellen is a very good treatment of this issue.
But I get back to what I consider the bigger problem - that of arguments and the use of data and theory to make a case. Until someone operationalizes high-level theory, it will not be useful in making rigorous and precise arguments, and will not contribute much to the development of a body of reliable scientific findings. I'm not holding my breath.
"They don't know how to put data together with theory to reach a rigorous conclusion about what likely happened in the past and why it happened." My girlfriend and I, both ABD archaeologists, have complained about the same thing in graduate education. No one seems to take this part of the research very seriously . . . but it is the most challenging aspect of rigorous research that is actually meaningful. Fact is, anyone can 'do theory.' The hard part is doing something with data.
@R Mahaney - I agree, and I think the lack of attention to this issue in graduate programs is widespread. There is a generation of young archaeologists and advanced students who cannot make good arguments.
Post a Comment