The speculation scale (the inverse of empirical adequacy)

ASU student Lisa Gallagher in our lab

I am posting this from the ASU lab at Teotihuacan in Mexico. I will be attending a conference on Teo sponsored by UNAM and Penn State over the next few days. On the trip down I read one of the worst articles I've read in a long time. I was surprised that the paper was accepted by a journal. It was published in a new journal, Economic Anthropology, whose standards have perhaps not risen to a level the editors would like. Did anyone review this paper?

This bad article got me thinking about the ratio between the scope or breadth of the claims made in a study, and the amount of data used. Works with a low ratio are often called "descriptive" studies. Works with a high ratio, on the other hand, contain little data, but make sweeping claims that go far beyond the data at hand. These are speculative studies, work that is poorly grounded. The paper I just read had a very high ratio, which is why I disliked it so much. Satisfying research in archaeology (and the social and historical sciences generally) usually falls in the mid-range of this scale.

I decided to create a graphic to illustrate this speculation scale. First I created nine data-to-interpretation schemes. These are combinations of three categories: low medium, and high amounts of data (blue circles), and low, medium and high levels of interpretation (red squares). I label these schemes from one  to nine, and arrange them on the speculation scale (see graphic).

Scheme 1 occupies the low, or descriptive, end of the scale. Those enamored of theory often dismiss such studies as "merely descriptive." But those of us who like to analyze data often find these studies useful as a source of data (that is, IF the data are adequately described and derive from rigorous methods, etc.). Schemes 2 and 3 are also descriptive in nature: the amount of data exceeds the amount of interpretation.

At the top of the scale, scheme 9 pertains to what I consider to be useless works. Who needs a bunch of speculation with little or no data? Unless the work is an editorial or opinion essay, or a work of pure theory, papers illustrating scheme 9 that purport to be empirical advances in fact contribute little to scholarship. How do these things get past reviewers and editors? But they do. Schemes 7 and 8 are also speculative in nature, although less flagrant than scheme 9. Quite a few studies in archaeology fit here. Most work in postprocessual and "social" archaeology probably correspond to schemes 7 and 8.

The schemes that fall in the middle of the speculation scale--schemes 4, 5, and 6--are those that seem most satisfying to most archaeologists. They are empirical studies based on real data that employ concepts and theories for explanation or interpretation. The amount of data is balanced by the amount of interpretation; the level of interpretation matches the amount of data.

*** ADDITION, Jan 30:  This graphic is probably a simpler way of showing the relationship:

*********

Obviously, this is a simplistic device for looking at the quality of publications. It says little about the quality of the data, or about the fit between interpretation and data. If you apply inappropriate models, your results will be suspect, even if the data and models are both of good quality on their own. But schemes like this can help us think about the adequacy of our models, interpretations, and explanations. If you have only a limited amount of data, then you should probably bypass big elaborate explanations and try to find something on a smaller scale. I know the postprocessualists and postmodernists will think I am being regressive here, but if archaeology is to be a science and generate reliable knowledge about the past, then we need to be able to match up our data with our interpretations/explanations. For more along these lines, see my series on archaeology as a science, starting with this post, or see my forthcoming paper:

Smith, Michael E.

2017?   Social Science and Archaeological Inquiry. Antiquity  (in press).

The field of economic anthropology has generally been relatively empirical in orientation, with good epistemology. I would think that a journal with that title would know better than to publish a paper whose score on the speculation scale (scheme 9) is so high. Ugh.