When big data are bad data

As archaeologists turn increasingly to the analysis of large, systematic databases, we need to confront an epistemological problem: How do we identify bad data, and what can we do about it? Economic historians and others are becoming consumers of archaeological data, and they are quick to jump on new databases. They seldom ask about the quality of the data, and this can result in sophisticated analyses of bad data. But, as we all know, “Garbage in, garbage out.”

I blogged about this a couple of years ago in reference to Tertius Chandler’s list of city sizes through history, from both archaeological and historical sources (Link is here). Those data (Chandler 1987) are considered shockingly bad and worthless by most historical demographers and historians. In technical terms, they may be "bullshit" (see my post on bullshit).

Yet some urban scholars merrily use the data for studies today. I consider this a real problem, and said so in my review of a manuscript for a journal titled “Scientific Data” (my blog post was an elaboration on that review). But data quality issues were evidently not as important to the authors and journal editors; the paper was published with only a few weak caveats about the data (Reba et al. 2016).

Another recent case focuses on the identification of the plague and other diseases in historical sources. This one focuses on a database compiled in 1975 by Jean-Noël Biraben of historical occurrences of the plague in France and the Mediterranean area. Specialists recognize numerous biases and problems with the basic data. But once the data were digitized, non-historians readily used them without question, leading to problematic results. The basic problem was pointed out by Jones and Nevell (2016), and elaborated on by Roosen and Curtis (2018).

Plague or not plague? From Jones/Nevell)

I highly recommend Roosen and Curtis (and thanks to my excellent colleague Monica Green for sending this my way). Their remarks parallel my views of the Chandler city-size data, but they do a better job of articulating the historiographical issues involved when subsequent scholars used these data (badly):

“When scholars fail to apply source criticism or do not reflect on the content of the data they use, the reliability of their results becomes highly questionable.” (p. 103).

Jones and Nevell deplore “the loosening of the rigorous standards of evidence and interpretation scientific researcher typically demand within their own disciplines.” (103)

Roosen and Curtis list three problems with recent analyses of the Biraben data (including a paper in PNAS): “First, reflection on the data collection process has been improper; second, what the data represent has not been recognized; and third, critique of the original sources has been inadequate. We argue that a critical consideration of any of these three elements would have led to the conclusion that the data set should not have been used at face value.” (105)

“However, through digitization and subsequent publication in a top-ranked journal, the 4-decade-old data set was imbued with a false aura of trustworthiness and the impression of being new historical research.” (104)

Joosen and Curtis make some recommendations for improving the systematic analysis of historical disease data. First, scholars need to employ basic historiographical techniques when they use data. That is, scholars need to subject data to source criticism, comparison of sources, contextual analysis. Second, hypothesis testing should be done on limited regions, rather than Europe as a whole (because of regional biases in Biraben’s material). And third, scholars should compile new, better databases. According to Monica Green, this is now in process with a new international collaborative project.

I think I’ve run out of steam; I’ll have to discuss Binford’s hunter-gatherer database in a separate post. This is a complicated and very troubling case, and I’ve been delaying writing about it for some time now, so I guess I’ll wait a bit longer.

So, what about big-data archaeology. Do we have any bad data? “Not me!” we all exclaim. Nevertheless, we all know that some of our colleagues are sloppier than others, and that some data are more reliable than others. Here are two suggestions.

First, take a historiographic approach to data, both your own and those of others. Historiography refers to the basic methods that historians use to evaluate their sources. Compare sources. Analyze possible biases in the sources. Analyze the context, purpose, and situation of the process of generating or gathering the data. Good metadata standards is a start, but I think we need to go further. Check out some of the methodological literature in history: (Henige 2005), (Hansen and Hansen 2016), (Kipping et al. 2014), and an old favorite of mine: (Fischer 1970); I still kick myself for not taking a class with Fischer when I was at Brandeis!

Second, archaeologists should work out methods of categorizing the reliability of our data and findings, and these should become part of the metadata of any database. There are basic methods of assessing the reliability of data and findings (6 and Bellamy 2012; Gerring 2012; Hruschka et al. 2004). The Intergovernmental Panel on Climate Change has worked out a useful system for coding the reliability of findings by different authors. They suggest evaluating independently the strength of the evidence, and the agreement among authorities (Adler and Hirsch Hadorn 2014).

These suggestions will have to be adapted to use with archaeological data. But they can help us avoid some of the problems that have arisen with the use of the faulty databases on city size and plague occurrence described above. As archaeologist rush ahead into the brave new world of big data, we should try to fix data problems sooner rather than later. We should try to ensure that big data are not bad data.

6, Perri and Christine Bellamy

2012 Principles of Methodology: Research Design in Social Science. Sage, New York.

Adler, Carolina E. and Gertrude Hirsch Hadorn

2014 The IPCC and Treatment of Uncertainties: Topics and Sources of Dissensus. Wiley Interdisciplinary Reviews: Climate Change 5 (5): 663-676.

Chandler, Tertius

1987 Four Thousand Years of Urban Growth: An Historical Census. St. David's University Press, Lewiston, NY.

Fischer, David Hackett

1970 Historians' Fallacies: Toward a Logic of Historical Thought. Harper, New York.

Gerring, John

2012 Social Science Methodology: A Unified Framework. 2nd ed. Cambridge University Press, New York.

Hansen, Bradley A. and Mary Eschelbach Hansen

2016 The historian's craft and economics. Journal of Institutional Economics 12 (2): 349-370.

Henige, David P.

2005 Historical Evidence and Argument. University of Wisconsin Press, Madison.

Hruschka, Daniel J., Deborah Schwartz, Daphne Cobb St. John, Erin Picone-Decaro, Richard A. Jenkins, and James W. Carey

2004 Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods 16 (3): 307-331.

Jones, Lori and Richard Nevell

2016 Plagued by doubt and viral misinformation: the need for evidence-based use of historical disease images. The Lancet Infectious Diseases 16 (10): e235-e240.

Kipping, Matthias, R. Daniel  Wadhwani, and Marcelo Bucheli

2014 Analyzing and Interpreting Historical Sources: A Basic Methodology. In Organizations in Time: History, Theory, Methods, edited by Marcelo Bucheli and R. Daniel Wadhwani, pp. 305-330. Oxford University PRess, New York.

Reba, Meredith, Femke Reitsma, and Karen C. Seto

2016 Data Descriptor: Spatializing 6,000 years of global urbanization from 3700 BC to AD 2000. Scientific Data 3 (160034).

Roosen, Joris and Daniel R. Curtis

2018 Dangers of Noncritical Use of Historical Plague Data. Emerging Infectious Disease journal 24 (1): 103-110.  http://wwwnc.cdc.gov/eid/article/24/1/17-0477. 

George Cowgill's database and my introduction to scientific archaeology

Here is a very short video that describes how I first got into the science of archaeology:

George had a database!

Ian Hodder says archaeology is bullshit. My reply: “Bullshit!”

In a remarkably bad short paper in the current SAA Archaeological Record, Ian Hodder makes a number of statements that equate to the claim that archaeology is bullshit (Hodder 2018). “Bullshit” is a term that refers to speech intended to persuade without regard for truth. Liars care about the truth and try to hide it; bullshitters don’t care whether their speech is true or false. Harry Frankfurt (1986, 2005) published the major works on bullshit, although antecedents can be found back to Plato and Orwell (1946 (1968)); see also Cohen (2002).

Hodder’s first dubious claim is that “the most important public value and function of archaeology is its role in place- and history-making” (p. 43). That is, archaeology is primarily about heritage, identity, and cultural achievement. It is about the present, not the past. Most archaeologists disagree with this. Archaeology is about the past. That is why we carry out excavations, surveys, artifact analyses and dating—to reconstruct and learn about human society in the past. Hodder’s first claim may be wrong and regressive, but it does not qualify as bullshit.

Hodder then gets to his main point: “much of archaeology uses the past to play out the contemporary preoccupations of dominant groups and to regurgitate the present in their interests … I have become tired of archaeologists just mirroring present concerns and theories” (p.43). The bad guys here are people like me, who study inequality, sustainability, or some of the other “grand challenges” we have identified for the discipline (Kintigh et al. 2014a; Kintigh et al. 2014b). Archaeologists go for headlines and not for local context, we are told; “This is what I mean by a post-truth archaeology or fake history.” (p. 44).

The concepts post-truth and fake news are typically applied to current affairs to refer to the kind of disregard for the truth captured in Frankfurt’s concept of bullshit. As Kathleen Higgins (2016) noted in Nature, “post-truth refers to blatant lies being routine across society, and it means that politicians can lie without condemnation … scientists and philosophers should be shocked by the idea of post-truth” (p. 9). So, Hodder is suggesting that people like me and my co-authors (Kohler et al. 2017), or the grand challenges crowd (Kintigh et al. 2014a; Kintigh et al. 2014b) are blatantly lying about the past. We are (knowingly, I guess) just projecting the concerns of the present—the “preoccupations of dominant groups”—back to the distant past.

I am not surprised that someone like Ian Hodder would characterize research by someone like me as post-truth and fake news. To make such an accusation, however, one must have workable concepts of science and truth in order to know that they have been violated. But, Hodder shows in this article (and elsewhere) that he has a faulty understanding of science. Like other post-processual archaeologists, Hodder thinks that science consists of discovering “universals that are singular in their unique law-like characteristics” (p. 43). In a recent paper in Antiquity (Smith 2017), I note how Hodder's post-processualist colleagues like Matthew Johnson (2010) criticize the concept of science in archaeology by employing a 50-year-old (outdated) definition of science and explanation. Science is not necessarily about universals and it is not necessarily about laws. It is about a rigorous search for evidence and explanation by constantly testing claims and hypotheses.

Contrary to Hodder’s assertion, those of us who use archaeological data to study phenomena such as sustainability, inequality, or political systems in the past do not adhere to the post-processualist caricature of science. Instead, we employ current concepts and epistemologies. These are aptly summarized by philosopher of science Daniel Little’s list of three epistemic features of science:

1. empirical testability

2. logical coherence

3. an institutional commitment to intersubjective processes of belief evaluation and criticism (Little 1995)

For additional statements of the nature of science in relation to archaeology and the other social sciences, see (Smith 2017), (Wylie 2000), (Gerring 2012:11), (Bunge 2011), (Little 2009). Or see some of my prior posts on this topic, including:

Science, social science, andarchaeology: Where do we stand?

Why is it important to strive for a morescientific archaeology?

Why is a scientific archaeology so hardto achieve?

Because Hodder has a faulty understanding of science, he has little basis for criticizing the scientific claims of other archaeologists. A look at the journals that my colleagues and I publish in (Science, Nature, Proceedings of the National Academy of Sciences, PLOS-One, etc.) shows that the “big question, big data approach” that Hodder dislikes (p.44) does indeed conform to contemporary scientific standards. So, just what standards are we violating that would warrant the labels post-truth and fake news? Hodder has none to offer.

I turn the tables here and characterize Hodder’s article as bullshit. He evidently does not know the nature of science, and thus his critique shows a disregard for the truthfulness or rigor of our work. His paper is post-truth, fake news, bullshit.

Bunge, Mario

2011 Knowledge: Genuine and Bogus. Science and Education 20: 411-438.

Cohen, G. A.

2002 Deeper into Bullshit. In Contours of Agency: Essays on Themes from Harry Frankfurt, edited by Sarah Buss and Lee Overton, pp. 321-339. MIT Press, Cambridge, MA.

Frankfurt, Harry

1986 On Bullshit. Raritan 6 (2): 81-100.

2005 On Bullshit. Princeton University Press, Princeton, NJ.

Gerring, John

2012 Social Science Methodology: A Unified Framework. 2nd ed. Cambridge University Press, New York.

Higgins, Kathleen

2016 Post-truth: a guide for the perplexed. Nature 540 (7631): 9.

Hodder, Ian

2018 Big History and a Post-Truth Archaeology? The SAA Archaeological Record 18 (5): 43-45.

Johnson, Matthew

2010 Archaeological Theory: An Introduction. 2nd ed. Blackwell, Oxford.

Kintigh, Keith W., Jeffrey Altschul, Mary Beaudry, Robert Drennan, Ann Kinzig, Timothy Kohler, W. Frederick Limp, Herbert Maschner, William Michener, Timothy Pauketat, Peter Peregrine, Jeremy Sabloff, Tony Wilkinson, Henry Wright, and Melinda Zeder

2014a        Grand Challenges for Archaeology. American Antiquity 79 (1): 5-24.  http://dx.doi.org/10.7183/0002-7316.79.1.5.

Kintigh, Keith W., Jeffrey H. Altschul, Mary C. Beaudry, Robert D. Drennan, Ann P. Kinzig, Timothy A. Kohler, W. Frederick Limp, Herbert D. G. Maschner, William K. Michener, Timothy R. Pauketat, Peter Peregrine, Jeremy A. Sabloff, Tony J. Wilkinson, Henry T. Wright, and Melinda A. Zeder

2014b        Grand Challenges for Archaeology. Proceedings of the National Academy of Sciences 122: 879-880.

Kohler, Timothy A., Michael E. Smith, Amy Bogaard, Gary M. Feinman, Christina E. Peterson, Aleen Betzenhauser, Matthew C. Pailes, Elizabeth C. Stone, Anna Marie Prentiss, Timothy Dennehy, Laura Ellyson, Linda M. Nicholas, Ronald K. Faulseit, Amy Styring, Jade Whitlam, Mattia Fochesato, Thomas A. Foor, and Samuel Bowles

2017 Greater Post-Neolithic Wealth Disparities in Eurasia than in North and Mesoamerica. Nature 551: 619-622.

Little, Daniel

1995 Objectivity, Truth, and Method. Anthropology Newsletter, American Anthropological Association Nov. 1995: 42.

2009 The Heterogeneous Social: New Thinking About the Foundations of the Social Sciences. In Philosophy of the Social Sciences: Philosophical Theory and Scientific Practice, edited by Mantzavinos Chrysostomos, pp. 154-178. Cambridge University Press, New York.

Orwell, George

1946 (1968)        Politics and the English language. In The collected essays, journalism and letters of George Orwell, pp. 127-140, vol. 4. Harcourt, Brace, Javanovich, New York.

Smith, Michael E.

2017 Social Science and Archaeological Inquiry. Antiquity 91 (356): 520-528.

Wylie, Alison

2000 Questions of Evidence, Legitimacy, and the (Dis)unity of Science. American Antiquity 65: 227-237.

George Cowgill, 1929-2018, a personal view

Some of the most vivid memories from my undergraduate days at Brandeis University are of my Friday afternoon meetings with George Cowgill. I took classes with George, he supervised my senior honors thesis on Teotihuacan, and I had a part-time job doing computer work for him. This was my first real encounter with research. I fell in love with archaeology, and I was discovering that I might be able to contribute some new knowledge to the field. At those meetings, I would go over what I had been working on with George. He was always positive and encouraging, but also critical. He would acknowledge what I had accomplished, but then show what more needed to be done. He had high expectations, and I wanted to live up to them.

I would leave his office full of excitement and drive. I can still picture the experience of running down the hill, toward my dorm, full of ideas. I was euphoric. Research was fun; not just fun, but captivating, intoxicating, wonderful. I was on to something, and I had some ideas of how to proceed. One of my goals as an undergraduate teacher is to try to kindle this kind of excitement in my students. When I see this kind of light in the eyes of a talented anthropology major, it makes me think of those Friday afternoons at Brandeis. It was George who got me excited about research, about Mesoamerica, about comparative early states, and about Teotihuacan.

I was heavily into math and science in high school. A summer NSF program in physics for high school students made me realize, however, that doing science was merely a puzzle for me, not something I was passionate about. I turned away from science, got involved in protests against the Vietnam war, and looked for a liberal arts college. When I discovered archaeology I started taking courses with George and other faculty. When it was time to think about a topic for my senior thesis, I was into mythology, symbolism, and human sacrifice (some of you are probably surprised here). I told George I wanted to work on a topic relating to religion at Teotihuacan, thinking I could study something gory like human sacrifice. He suggested I compare the artifact assemblages of temples and residences, using the Teotihuacan database. I was crestfallen - this didn't sound sexy or exciting. But it was exactly what I needed at the time. It was just the kind of project to turn me in the direction of rigorous, scientific, quantitative research. And I haven't looked back since.

The Millon / Cowgill map of Teotihuacan

George arranged for my first fieldwork project in Mexico. He made some phone calls, and arranged for a fellow student and I to work with William Sanders on the last season of the Basin of Mexico archaeological survey project. I also spent some time in the lab of the Teotihuacan Mapping Project to learn something about the ceramics and artifacts. That summer, I fell in love with Mexico: with the archaeology, the people, the food, the music.

From my senior honors thesis

After I graduated in 1975, I saw George on and off over the years. We crossed paths at meetings, and I usually took my field crews up to Teotihuacan. We'd see George in the nice new lab he had built (with NSF funds). I followed George's work, of course. When he retired from Arizona State University, I applied for the central Mexicanist position at ASU. I had not kept up a high level of statistical and quantitative methods, and I was worried that I would disappoint George (and perhaps the search committee as well). But I got the job. So, I was hired to replace my old undergrad professor! How cool is that? It is one of the really satisfying events of my career.

From my hire at ASU in 2005, George and I were colleagues. Although retired, George came in to campus regularly. We exchanged papers and ideas. When some of us formed a transdisciplinary research group on urban neighborhoods, George was a regular participant. We all benefited from his expertise, judgment, and humor. George would always attend parties. He'd usually just stay in one chair the whole time, but students and others would circulate and talk with him. He became very active on Facebook. As George started to slow down, the ASU administration appointed me Director of the lab. George was helpful in getting me oriented, always available for questions and advice. He also established an endowment to help fund the lab's operations.

The ASU Teotihuacan lab today

Two years ago, Cindy and I helped George's niece Karen Cowgill by supervising the organization and clearing of things at George's house in Tempe. He was in an assisted living facility, where he would remain. I wanted to make sure we kept track of any research materials he had at home and in his office. Karen hired 3 anthropology majors to help. They discovered George had saved his old Brandeis gradebooks in a box in the garage. Oh, let's see how well Dr. Smith did as a student! Luckily, my honor was upheld - I got grades of "A" in the two classes included in the grade book.

George Cowgill's scientific and professional contributions are many, and I'm sure they will be reviewed in some academic obituaries. I just want to close with some of the ways my own thinking and intellectual development were shaped by my undergraduate experiences working with George.

Artifact boxes in George's lab

I thank George for encouraging in me a rigorous, scientific approach to archaeology. George pioneered quantitative and computer approaches in archaeology. He got me using cluster analysis and discriminant function analysis. At the time, these were exotic methods, not available on the Brandeis mainframe. George let me use his account on the Harvard mainframe for my thesis research. At some point I'll write up some of my humorous stories of doing random sampling in odd situations, and the Harvard computer center was the setting for one of them. I DID manage to get a random sample that time; I persevered largely because I didn't want to disappoint George. Discriminant analysis became one of my favorite multivariate methods, used in the several quantitative ceramic seriations I've worked on (George also contributed to the development of seriation methods).

I also took from George a love of comparative study of early states. I was always fascinated with a long, vertical chart on the wall of his Brandeis office. It showed the population sizes of prominent cities, ancient and modern, plotted on green log paper. I was thrilled to come across that very chart, faded and scuffed, when clearing out George's office. I haven't yet found a place for it on my wall.

And,  of course, there is Teotihuacan. It was clearly my first love in Mesoamerican archaeology. I went on to devote my career to Aztec sites. Returning to Teotihuacan when I took over the lab in 2015, I felt some of that old excitement from my Brandeis days. My experiences with Aztec sites and comparative urbanism allowed me to view Teotihuacan from a new perspective, and I think George appreciated this. In more ways than one, I would not be where I am as a scholar and a person, if not for George Cowgill. I will miss him deeply, as will many others.

How archaeology is distorted by Science magazine and the National Geographic Society

The public has a lot of interest in archaeology, and new finds and discoveries are often in the news. Sometimes the press reports outlandish, nonsense claims, and sometimes it reports rigorous and important claims. Many archaeologists ignore such press coverage (good and bad), and many of us engage with it. If there are elements of the press that distort archaeology, we may or may not want to deal with this. I often ask myself if it is worth the time and effort to try to correct some misleading claim going around social media and the internet. In a very real sense, this kind of thing can be viewed as separate and apart from the actual process of scientific research. I can do my archaeology just fine without worrying about whether the latest "news" about an ancient monkey god is a pile of baloney or not.

But there are other distortions of archaeology that are more insidious and more troubling. These come from well-established, serious institutions whose missions include furthering research and knowledge in many fields, including archaeology. I am thinking here of Science magazine and the National Geographic Society. Science publishes archaeology, and thus promotes the scholarly work of archaeologists while showcasing our work to a wider audience beyond archaeology. This is good. But the kinds of archaeology article they publish is a biased and distorted sample of current scientifically-inclined archaeological research. Thus, Science distorts archaeology.  The National Geographic Society funds archaeology, and also publicizes archaeology in its website, TV shows, and magazine. Again, this is good. But, again, the archaeology promoted by NGS is often a distorted view of the past, emphasizing spectacular and mysterious finds, often to the point of severe distortion. This is done for commercial gain.

Science Magazine

The review process for Science is a clear example of scholars and individuals who are not archaeologists making decisions about what archaeology is worthy of publication in Science, and those decisions distort our field. The single most popular post in the 11-year history of this blog -- by far -- is one from 2012 called "Rejected by Science" (over 100,000 hits, when most of the posts get well under 5,000 hits). So, let me quote from that post:

 If you pay attention to the journal, you will know that they tend to favor high-tech methods, archaeometry, fancy quantitative methods, and reports about "the earliest" this or that. While I can only recall one or two papers in Science that I thought were incompetent (a much better record than most archaeology journals, some of which are full of incompetent articles), their selection of archaeology papers is definitely biased in a certain direction. I think one way of expressing this might be that Science publishes archaeology articles that will appeal on methodological grounds to non-archaeological scientists. My guess is that papers that are more synthetic or less methods-heavy don't make it through the initial review (which is done by non-archaeological scientists).

I have been rejected by Science at least four times (and about 3 times by Nature, and a couple of times by PNAS). I've been rejected by some of the finest journals in the world! So, maybe my complaint is just sour grapes. But I have had many colleagues over the years express agreement with my sentiments as quoted above. Papers are first skimmed by a high-level reviewer, none of whom is an archaeologist. If they think it might be interesting, the manuscript is sent out for review. Most papers are dropped at this stage. I must say, Nature does the same thing, but quicker. I think one of my rejections from Nature took only a couple of hours!

The fact that papers are initially screened by someone who is NOT an expert in the subject matter of the paper is a common critique of the Science review process. Indeed, one of the ways that PLOS-One touts their rigor is their claim that all papers will be reviewed by subject-matter experts.

This review process, and its outcomes, clearly distorts the archaeological content published in the journal Science. Paper are not judged by archaeologists, and only a narrow range of types of archaeological papers get published. Check out my older on issues with Science and Nature:

"Rejected by Science!"  (2012), initial post

"Rejected by Science, Yet Again!" (2012)

"Problems with Science and Nature"  (2013)

National Geographic Society

The publicity put out by the National Geographic Society provides another case where the nature of the archaeology that is funded and reported is often distorted. But instead of a bias within science, as in the case above, this is a commercial bias that favors sensationalism, mystery, and click-baiting themes. The NGS website may make sensationalist claims, and these get taken up by the mainstream media. The Guardian often seems to swallow them, hook, line, and sinker. The latest example is the claim that archaeologist Chris Fisher has discovered an ancient city in Mexico, using LiDAR, that "had as many buildings as Manhattan," to quote the Guardian. Some Mexican and French archaeologists objected in the Mexican paper, El Pais. They claimed that a city that big could not possibly have existed, because it was not mentioned in the historical sources! Give me a break, that is just silly. The city (Angamuco) IS large, but much of the LiDAR has not been ground-truthed yet, so this is a speculative claim.

Another recent case was the announcement that a new huge city was found below the jungle in Guatemala by LiDAR! It was accompanied in the press by image of Tikal. See my prior post on this. The National Geographic Society press release is mostly hype and speculation, which then got worse as other media outlets picked up the story. My purpose here is not to question hype or expose silly claims. Rather, I want to emphasize that the nature of the archaeological finds that get promoted are in the hands of an institution outside of archaeology, one that does a lot of good through funding and publicizing. Maybe we should give NGS a break, since they do a lot of good for archaeology. But I am more inclined to question their methods and goals. They are commercializing archaeology, and using our hard work to make a buck. That might be ok, if they were more objective in their reporting. I did get some funds from NGS back around 1990, but I'm not sure I would accept their funding now. Who knows what nonsense they could create from my research.

More hype than archaeology

If you want an extended example of NGS nonsense and hype, check out the story of the "Lost city of the Monkey God." NGS-funded "explorers" tramped around the Honduran jungle and came out with outrageous claims about finding a lost city. Never mind that the site in question was well known to archaeologists and had been studied earlier. A whole story of intrepid explorers and fantastic discoveries was generated. The resulting book has become a best-seller. Please check out these sources for some objective, archaeologically-appropriate writing about this episode:

blog post by Rosemary Joyce from 2012

another post by Joyce, 2015

article in Smithsonian magazine about the episode (2015)

An open letter to NGS signed by a bunch of archaeologists that lists all the problems with the activities and their reporting.

These archaeologists are so discouraged that they feel that any additional harping will only increase sales of the book by Preston. I hope I am not contributing to the hype here.

Apart from the scientific and scholarly nonsense of the Monkey God episode, I find it discouraging for the same reason as the reviewing procedures used by Science: archaeology is being distorted by non-archaeological institutions that purportedly exist, in part, to promote and improve archaeology.

Why I am skeptical about the new Maya LiDAR results from NGS

New Tikal LiDAR map

I am skeptical about the hype surrounding the recent press release from the National Geographic Society about the new findings of LiDAR survey in the Maya region of northern Guatemala. I have no reason to question the quality of the LiDAR survey, or its potential usefulness for understanding aspects of ancient Maya society in this region. Rather, I question two aspects of the way these new findings have been portrayed, both in the NGS press release and in the journalism that has resulted from the find. (1) This is portrayed as revealing brand-new ideas, when in fact earlier LiDAR work had very similar results; and (2) The work is portrayed as a major scientific discovery, when in fact it is only the first step of a process, the end result of which will be (one hopes) some major scientific discoveries.

LiDAR is a relatively new airborne remote sensing technology that permits detailed mapping of the surface of the earth at a detailed scale. It is far superior to earlier forms of satellite or airplane mapping in that LiDAR can penetrate dense vegetation. It is ideal for the Maya lowlands, where the jungle vegetation hinders traditional mapping. Wherever it has been applied, in the Maya area, the result is the identification of many new houses and features of the built environment. (1) This is my first misgiving: the lack of acknowledgement that Mayanists have been working with LiDAR for more than seven years (Chase et al. 2012; Chase et al. 2011).

The NGS story has breathless quotes about how suddenly we know about many new features and structures on the Maya landscape. Well, that is what LiDAR does in the Maya lowlands. It finds many more features than archeologists knew about previously. If archaeologists are surprised about this, they just haven’t looked at the prior work, both in the Maya area (Brown et al. 2016; Chase et al. 2014b; Chase et al. 2014a; Chase et al. 2012; Chase et al. 2011; Chase et al. 2016; Chase 2016; Chase and Weishampel 2016; Ebert et al. 2016; Prufer et al. 2015; Von Schwerin et al. 2016; Yaeger et al. 2016), in other parts of Mesoamerica (Fisher and Leisz 2013; Rosenswig et al. 2015; Rosenswig et al. 2013), and particularly at Angkor in Cambodia (Evans et al. 2013; Hanus and Evans 2016).

 

Lidar-identified small reservoirs at Cacacol. Chase 2016.

One difficulty with LiDAR data is that while it is easy to see large structures like pyramids in the output data, small features such as houses or agricultural fields are more difficult to pick out. They often require a combination of intensive, time-consuming searching by eye, and sophisticated custom computer algorithms that can pinpoint such features automatically. For example, my student, Adrian Chase, analyzed LiDAR data to identify small residential-level reservoirs at the Maya city of Caracol (Chase 2016). In areas that had been mapped previously by traditional methods, Adrian’s algorithm identified 25 times the number of small reservoirs at the site! These did not stand out on the LiDAR landscape like dropped pins in Google-Maps. They had to be painstakingly identified.

As far as I can tell, the intensive phase of analysis has not yet been carried out (or is not reported in this press release). It is easy to use LiDAR to find a bunch of new features and make a pretty map. But the next two steps are more difficult. For the first step, the archaeologist has to analyze the data—staring at maps and applying algorithms—so that one can be confident that most of the relevant small features have been identified. The pretty color maps one sees in all the press accounts are not the only way to portray spatial data in LiDAR; often other visualization methods are more useful. Adrian was able to identify all those small reservoirs only because he did two things: he spent countless hours staring at the output, and he applied custom computer algorithms to the data to identify the features. There is no indication that archaeologists have carried out this intensive level of analysis of the new Guatemalan data.

 

3 LiDAR visualizations. Chase 2016

A second crucial step is to analyze the results quantitatively and spatially to construct population estimates and study the on-the-ground patterning in settlement data. The NGS article subtitle says there were “millions more people than previously thought.” The report has this quote:

“Most people had been comfortable with population estimates of around 5 million,” said [Francisco] Estrada-Belli, who directs a multi-disciplinary archaeological project at Holmul, Guatemala. “With this new data it’s no longer unreasonable to think that there were 10 to 15 million people there—including many living in low-lying, swampy areas that many of us had thought uninhabitable.”

It will take quite a bit of analysis to turn this quick preliminary suggestion into rigorous population estimates for settlements and regions. These additional steps—technical application of algorithms, lots of staring at screens, and then quantification and calculation—are only beginning for the Maya lowlands (Chase 2016; Chase and Weishampel 2016; Ebert et al. 2016), and there is no sign that they have been accomplished for the new Guatemalan LiDAR results.

 

LiDAR of central Caracol. Chase et al 2011.

So, what is my beef? The new results are just in, and the analysis is probably only starting. This is the normal process of science. (2) My second misgiving is the idea—promoted by NGS, by the people interviewed in the article, and by secondary articles in the media—that archaeological advances consist of discoveries in the field. Yes, the fieldwork is essential. But without an often lengthy period of analysis, one typically cannot know the meaning or importance of the finds.

There is a kind of archaeology where the main discovery is made in the field. If one is looking for the tomb of a king or noble, and one finds it, that may be the essential defining moment of discovery. But I pursue another kind of archaeology. I have spent my career on the archaeology of Aztec provincial households. When I dig up another house or trash midden, it seems pretty much the same as countless I and others have excavated. They are pretty boring, I have to admit. But once I have spent months or years studying the artifacts, quantifying them, sending off samples of technical analyses, only then do I make my discoveries. When I argue that this household was well-off and that one was poor, or when I argue that conquest by the Aztec empire had little effect on local people, these are my discoveries. They rely on extensive analyses of artifacts. I had no idea about these things at the time of excavation. I discuss this issue—what is the real moment of discovery?—in more detail in my recent book (Smith 2016).

 

Social interpretations at Yautepec were based on study of 1 million potsherds

When one focuses almost exclusively on the actual uncovering of a find during fieldwork (for an excavation), or on the initial pretty maps of a LiDAR survey—before the hard work of analysis is done—one is distorting the scientific significance of our work. Will NGS have a big feature when the archaeologists involved actually publish a revised population estimate for northern Guatemala, or when they can quantify the amount of construction in rural vs urban areas?
We'll see.

A kind of archaeology based on extensive analysis

How can one spot a finding that seems spectacular but is actually a preliminary find, not yet analyzed, from a finding based on proper analysis and interpretation? Peer-review publication is the primary way to do this. The NGS piece was based entirely on interviews, not on a paper that has been peer-reviewed and accepted by a scientific journal.

Claims that LiDAR will revolutionize the study of Maya settlement and demography may very well be correct, but it is too soon to tell. The Guatemalan LiDAR has reached the stage of preliminary findings and pretty maps, but not the stage of solid architectural, demographic, and social findings. I look forward to the scientific results. I don’t care if they are an internet sensation; I’d rather see them published in a journal.

References

Brown, M. Kathryn, Jason Yaeger, and Bernadette Cap

2016 A Tale of Two Cities; LiDAR Survey and New Discoveries at Xunantunich. Research Reports in Belizean Archaeology 13: 51-60.

Chase, Adrian S. Z.

2016 Beyond Elite Control: Residential Reservoirs at Caracol, Belize. WIREsWater 3 (6): 763-797.

Chase, Adrian S. Z. and John F. Weishampel

2016 Water Capture and Agricultural Terracing at Caracol, Belize as Revealed through Lidar and GIS. Advances in Archaeological Practice 4 (3): 357-370.

Chase, Arlen F., Diane Z. Chase, Jaime J Awe, John F. Weishampel, Gyles Iannone, Holley Moyes, Jason Yaeger, and M. Kathryn Brown

2014a        The Use of LiDAR in Understanding the Ancient Maya Landscape. Advances in Archaeological Practice 2 (3): 208-221.

Chase, Arlen F., Diane Z. Chase, Jaime J. Awe, John F. Weishampel, Gyles Iannone, Holly Moyes, Jason Yaeger, Kathryn Brown, Ramesh L. Shrestha, William E. Carter, and Juan Fernandez Diaz

2014b        Ancient Maya Regional Settlement and Inter-Site Analysis: The 2013 West-Central Belize LiDAR Survey. Remote Sensing 6: 8671-8695.

Chase, Arlen F., Diane Z. Chase, Christopher T. Fisher, Stephen J. Leisz, and John F. Weishampel

2012 Geospatial revolution and remote sensing LiDAR in Mesoamerican archaeology. Proceedings of the National Academy of Sciences 109: 12916-12921.

Chase, Arlen F., Diane Z. Chase, John F. Weishampel, Jason B. Drake, Ramesh L. Shrestha, K. Clint Slatton, Jaime J. Awe, and William E. Carter

2011 Airborne LiDAR, Archaeology, and the Ancient Maya Landscape at Caracol, Belize. Journal of Archaeological Science 37: 387-398.

Chase, Arlen F., Kathryn Reese-Taylor, Juan C. Fernandez-Diaz, and Diane Z. Chase

2016 Progression and Issues in the Mesoamerican Geospatial Revolution: An Introduction. Advances in Archaeological Practice 4 (3): 219-231.

Ebert, Claire E., Julie A. Hoggarth, and Jaime J. Awe

2016 Integrating Quantitative Lidar Analysis and Settlement Survey in the Belize River Valley. Advances in Archaeological Practice 4 (3): 284-300.

Evans, Damian H., Roland J. Fletcher, Christophe Pottier, Jean-Baptiste Chevance, Dominique Soutif, Boun Suy Tan, Sokrithy Im, Darith Ea, Tina Tin, Samnang Kim, Christopher Cromarty, Stéphane De Greef, Kasper Hanus, Pierre Bâty, Robert Kuszinger, Ichita Shimoda, and Glenn Boornazian

2013 Uncovering archaeological landscapes at Angkor using lidar. Proceedings of the National Academy of Sciences 110: 12595-12600.

Fisher, Christopher T. and Stephen J. Leisz

2013 New Perspectives on Purapécha Urbanism through the Use of LiDAR at the Stie of Angamuco, Mexico. In A Primer on Space Archaeology: In Observance of the 40th Anniversary of the World Heritage Convention, edited by D.C. Comer, pp. 191-202. SpringerB riefs in Archaeology, vol. 5. Springer, New Yokr.

Hanus, Kasper and Damian Evans

2016 Imaging the Waters of Angkor: A Method for Semi‐Automated Pond Extraction from LiDAR Data. Archaeological Prospection 23 (2): 87-94.

Prufer, Keith M., Amy E. Thompson, and Douglas J. Kennett

2015 Evaluating airborne LiDAR for detecting settlements and modified landscapes in disturbed tropical environments at Uxbenká, Belize. Journal of Archaeological Science 57: 1-13.

Rosenswig, Robert M., Ricardo López-Torrijos, and Caroline E. Antonelli

2015 Lidar data and the Izapa polity: new results and methodological issues from tropical Mesoamerica. Archaeological and Anthropological Sciences 7 (4): 487-504.

Rosenswig, Robert M., Ricardo López-Torrijos, Caroline E. Antonelli, and Rebecca Mendelsohn

2013 LiDAR Mapping and Surface Survey of the Izapa State in the Tropical Piedmont. Journal of Archaeological Science 40: 1493-1507.

Smith, Michael E.

2016 At Home with the Aztecs: An Archaeologist Uncovers their Domestic Life. Routledge, New York.

Von Schwerin, Jennifer, Heather Richards-Rissetto, Fabio Remondino, Maria Grazia Spera, Michael Auer, Nicolas Billen, Lukas Loos, Laura Stelson, and Markus Reindel

2016 Airborne LiDAR acquisition, post-processing and accuracy-checking for a 3D WebGIS of Copan, Honduras. Journal of Archaeological Science: Reports 5: 85-104.

Yaeger, Jason, M Kathryn Brown, and Bernadette Cap

2016 Locating and dating sites using Lidar survey in a mosaic landscape in Western Belize. Advances in Archaeological Practice 4 (3): 339-356.