The speculation scale (the inverse of empirical adequacy)

ASU student Lisa Gallagher in our lab

I am posting this from the ASU lab at Teotihuacan in Mexico. I will be attending a conference on Teo sponsored by UNAM and Penn State over the next few days. On the trip down I read one of the worst articles I've read in a long time. I was surprised that the paper was accepted by a journal. It was published in a new journal, Economic Anthropology, whose standards have perhaps not risen to a level the editors would like. Did anyone review this paper?

This bad article got me thinking about the ratio between the scope or breadth of the claims made in a study, and the amount of data used. Works with a low ratio are often called "descriptive" studies. Works with a high ratio, on the other hand, contain little data, but make sweeping claims that go far beyond the data at hand. These are speculative studies, work that is poorly grounded. The paper I just read had a very high ratio, which is why I disliked it so much. Satisfying research in archaeology (and the social and historical sciences generally) usually falls in the mid-range of this scale.

I decided to create a graphic to illustrate this speculation scale. First I created nine data-to-interpretation schemes. These are combinations of three categories: low medium, and high amounts of data (blue circles), and low, medium and high levels of interpretation (red squares). I label these schemes from one  to nine, and arrange them on the speculation scale (see graphic).

Scheme 1 occupies the low, or descriptive, end of the scale. Those enamored of theory often dismiss such studies as "merely descriptive." But those of us who like to analyze data often find these studies useful as a source of data (that is, IF the data are adequately described and derive from rigorous methods, etc.). Schemes 2 and 3 are also descriptive in nature: the amount of data exceeds the amount of interpretation.

At the top of the scale, scheme 9 pertains to what I consider to be useless works. Who needs a bunch of speculation with little or no data? Unless the work is an editorial or opinion essay, or a work of pure theory, papers illustrating scheme 9 that purport to be empirical advances in fact contribute little to scholarship. How do these things get past reviewers and editors? But they do. Schemes 7 and 8 are also speculative in nature, although less flagrant than scheme 9. Quite a few studies in archaeology fit here. Most work in postprocessual and "social" archaeology probably correspond to schemes 7 and 8.

The schemes that fall in the middle of the speculation scale--schemes 4, 5, and 6--are those that seem most satisfying to most archaeologists. They are empirical studies based on real data that employ concepts and theories for explanation or interpretation. The amount of data is balanced by the amount of interpretation; the level of interpretation matches the amount of data.

*** ADDITION, Jan 30:  This graphic is probably a simpler way of showing the relationship:

*********

Obviously, this is a simplistic device for looking at the quality of publications. It says little about the quality of the data, or about the fit between interpretation and data. If you apply inappropriate models, your results will be suspect, even if the data and models are both of good quality on their own. But schemes like this can help us think about the adequacy of our models, interpretations, and explanations. If you have only a limited amount of data, then you should probably bypass big elaborate explanations and try to find something on a smaller scale. I know the postprocessualists and postmodernists will think I am being regressive here, but if archaeology is to be a science and generate reliable knowledge about the past, then we need to be able to match up our data with our interpretations/explanations. For more along these lines, see my series on archaeology as a science, starting with this post, or see my forthcoming paper:

Smith, Michael E.

2017?   Social Science and Archaeological Inquiry. Antiquity  (in press).

The field of economic anthropology has generally been relatively empirical in orientation, with good epistemology. I would think that a journal with that title would know better than to publish a paper whose score on the speculation scale (scheme 9) is so high. Ugh.

Some consequences of viewing archaeology as a social science

“Making warranted inferences is the whole point and the only point of doing social research.”

(6 and Bellamy 2012:14)

If one views archaeology as a social science, then the poor record of many archaeologists in making arguments is a serious professional problem. I point out problems with a number of common archaeological approaches to argumentation in a recent paper (Smith 2015); see also Smith (2011). I have just started reading a book (6 and Bellamy 2012) that covers similar material as my article, but on steroids. That is, the book is an extended treatment of the process of argumentation in the social sciences. Although I am only on chapter 2 now, the book has stimulated me to think about the implications of faulty argument techniques for a social-scientific view of the discipline of archaeology. If you have other views of archaeology--perhaps considering it a branch of the humanities, or as consisting of practical fieldwork and not a scholarly discipline--then the following argument does not apply.

What are the social sciences?

Here are a few quotes that describe what social science is. These are mainstream social science perspectives that--in my opinion--are congenial to much of archaeology.

• “Social science is about human beings and what goes on in their institutions and interactions .... Social science studies all levels from the family to the global society, and how the various levels affect each other” (Steuer 2002:5).

Ragin and Amoroso (2011:33-56) list the following as the main research goals of research in the social sciences:

1. Identifying general patterns and relationships
2. Testing and refining theories
3. Making predictions.  [But see Lieberson and Lynn (2002) for a good discussion of sociology as an observational, not a predictive, discipline. Ditto for archaeology.]
4. Interpreting culturally or historically significant phenomena
5. Exploring diversity
6. Giving voice
7. Advancing new theories

What is the purpose of the social sciences?

•  “The main structural features of what society can be like in the next generation are already given by trends at work now. Humanity’s freedom of maneuver lies within the framework created by its history. Social scientists and allied scholars could help to widen the area of choice by analyzing the historical trends that now limit it. They could show, impartially, honestly, and free from the special pleadings of government and vested interests, the range of possible alternative and the potentialities for effective action. Such has been, after all, the aim of inquiry into human affairs in free societies since the Greeks” (Moore 1958:159).

•  “The purpose of social science, let us say, is to help citizens and policymakers better understand the world, with an eye to changing that world. Social science ought to provide useful answers to useful questions" (Gerring 2012:396)

What do social scientists have to do to reach the goals identified by Ragin and Amoroso, and to accomplish the social purposes described by Barrington Moore and John Gerring? The mainstream social science answer to this question is that social scientists need to do two things:  First, conduct research using scientific methods; and, second, reach conclusions based on rigorous forms of scientific argumentation (any of the methods textbooks in the social sciences will elaborate on this claim; e.g., Abbott 2004; Ragin and Amoroso 2011; Gerring 2012). In the words of John Gerring, 

• “The willful avoidance of scientific methodology has doleful long-term consequences for social science, and for those who would see social science playing a role in the transformation of society" (Gerring 2012:399).

My focus here is on the second requirement of social research: good argumentation (Smith 2015). The epigram, a quote from 6 and Bellamy, strikes me as crucial:

• “Making warranted inferences is the whole point and the only point of doing social research.”

Then were does this leave those many archaeologists who are satisfied to quote a bunch of abstract, philosophical social theory, then describe their data, and end up with a bunch of speculative conclusions? These authors are not making what 6 and Bellamy call "warranted inferences." To my mind, this non-rigorous form of argumentation is harmful to an archaeology that strives to be part of the social sciences. It may be okay for a postmodern archaeology, or for an archaeology that eschews science to focus on literary theory or some other kind of humanities epistemology. But to my way of thinking, if we can't make rigorous, warranted arguments, then why bother doing archaeology? 

References

6, Perri and Christine Bellamy
2012    Principles of Methodology: Research Design in Social Science. Sage, New York.

Abbott, Andrew
2004    Methods of Discovery: Heuristics for the Social Sciences. Norton, New York.

Gerring, John
2012    Social Science Methodology: A Unified Framework. 2nd ed. Cambridge University Press, New York.

Lieberson, Stanley and Freda B. Lynn
2002    Barking up the Wrong Branch: Scientific Alternatives to the Current Model of Sociological Science. Annual Review of Sociology 28: 1-19.

Moore, Barrington, Jr.
1958    Political Power and Social Theory. Harvard University Press, Cambridge, MA.

Ragin, Charles C. and Lisa M. Amoroso
2011    Constructing Social Research: The Unity and Diversity of Method. 2nd ed. Sage, Thousand Oaks, CA.

Smith, Michael E.
2011    Empirical Urban Theory for Archaeologists. Journal of Archaeological Method and Theory 18: 167-192.

Smith, Michael E.
2015    How can Archaeologists Make Better Arguments? The SAA Archaeological Record 15 (4): 18-23.

Steuer, Max
2002    The Scientific Study of Society. Kluwer, New York.

How to make stronger arguments

 

My paper "How can archaeologists make better arguments?" has recently been published in the SAA Archaeological Record. While this is just a short paper in a non-peer-reviewed newsletter, I think it is one of my more important publications. Many aspects of contemporary archaeology -- particularly the archaeology of complex societies -- annoy and depress me. I use this blog to blow off steam about issues of data, theory, empirical adequacy, quality control, and such. I have come to the realization much of my dissatisfaction with publishing today revolves around the low quality of the empirical arguments in much of the literature today. People draw conclusions that are not warranted by the data, and then those results become enshrined as facts for future research and publishing (they are published! by a well-known archaeologist! they must be true!). Yet they are not facts at all; they are speculations with little empirical content.

I did some sniffing around the internet for my graduate theory seminar last year. I found that the topic of making arguments, like other epistemological topics, was rarely covered in graduate theory classes in the U.S. This surprised me at first, but then it made sense. If graduate students weren't getting training in how to make a rigorous argument -- how to test models with data, how to use theory to construct causal arguments -- then this helps explain the sorry state of archaeological argument. The alternative to rigorous empirical epistemology is philosophical, abstract theory, without much empirical content.

It seemed to me that rather than just ranting and raving about theory and epistemology in this blog, I should write a clear and direct guide to empirical arguments in archaeology. Some of the intellectual background is provide in my 2011 urban theory paper, but the new article is short and direct. So, if you haven't seen it yet, please download the paper and read it. And then please follow the advice. I want you to have a clearer view of archaeological epistemology, and I want you to make better arguments. This is not altruism on my part; instead, this advice comes from my distress at the state of archaeological knowledge today. If everyone were to make better arguments, the whole field would be vastly improved and perhaps I could stop being embarrassed when I have to explain away sloppy archaeological papers to my colleagues in other disciplines.

Smith, Michael E.    2015    How can Archaeologists Make Better Arguments? The SAA Archaeological Record 15(4): 18-23.

Smith, Michael E.    2011    Empirical Urban Theory for Archaeologists. Journal of Archaeological Method and Theory 18:167-192.

How to make a weak argument

Suppose you are writing up some archaeological results. You will be making a bunch of arguments--statements that draw on data and theory to come to some conclusion of interest. Most works contain a number of arguments, often at different levels. For example you make claim that you found 41 pieces of obsidian in the lowest level and only 14 in the uppermost level. This is an argument, but it is not a particularly interesting one. You may later make a more interesting argument suggesting that the decline in obsidian was due to changing commercial routes that now avoided your site, or perhaps you will argue that the decline came from a reduction in blood-letting rituals that employed obsidian blades.

Now, suppose you decide that you want to make a weak argument that few of  your colleagues will find convincing. While I am of course being sarcastic here, as I was in my post, "How to give a bad conference paper", this is a serious point. Why? Because it often seems that archaeologists must at some level be making this decision. They make weak arguments. So, the purpose of this post is to help them out by reminding them of tips and tricks to make bad arguments. I will mostly provide links to past blog posts where I discuss these issues.

(1) Use analogy incorrectly.

Do not develop a formal argument by analogy, based on a sample of source examples and carefully extrapolated to your archaeological data. Ignore Lewis Binford's suggestion to treat an analogy as a hypothesis to test, and whatever you do, make sure you avoid Alison Wylie's brilliant and definitive discussion of the role of analogy in archaeology. Cherry-pick one analogical case from somewhere in the world and claim that it supports your case.

You could check out my previous post on this topic, although be warned that it is a reverse argument: it assumes that you might want to make strong arguments and use analogy well.

(2) Make post-hoc interpretations.

Don't bother to set up initial hypotheses or expectations. Who knows what you fill find when you dig into the ground, anyway? Do your fieldwork or lab analysis, then scratch your head and try to dream up a nice-sounding interpretation. Slap some currently fashionable idea onto your data, and voila, you are done.

You could check out my earlier post on post-hoc arguments, or the one on trying to prove that you are wrong.

(3) Use empty citations to back up your shoddy scholarship.

Don't use citations to other works to supply data and cases that provide a foundation for your arguments. Instead, cite sources that that have no empirical data, but rather offer opinions and speculations that agree with your argument. Avoid citing studies with data that go against your views or models; instead cite those that agree with your ideas but lack any data. These are called empty citations.

You can check my prior post on this topic, and please follow the links there to Ann-Wil Harzing's original discussion of empty citations. Oops, I am being straight here, not sarcastic.


I find it really depressing that the archaeological literature (particularly the archaeology of complex societies) is so full of weak arguments. This acts to prevent the development and accumulation of reliable archaeological findings, which impedes the empirical advancement of our field. The sloppy use of analogy, post-hoc interpretations, and empty citations are all part of the picture. We need to get our act together. If you have not read and carefully studied Wylie (1985), you should do that immediately. And then check out chapter 7 of Booth et al (2008). Check out some of the methodological works from the social scientists I cited in my previous post. And finally, read my article on this topic (once I get around to writing it.......)

Booth, Wayne C., Gergory G. Colomb and Joseph M. Williams  (2008)  The Craft of Research. 3rd ed. University of Chicago Press, Chicago.

Wylie, Alison  (1985)  The Reaction Against Analogy. Advances in Archaeological Method and Theory 8:63-111.