When big data are bad data

As archaeologists turn increasingly to the analysis of large, systematic databases, we need to confront an epistemological problem: How do we identify bad data, and what can we do about it? Economic historians and others are becoming consumers of archaeological data, and they are quick to jump on new databases. They seldom ask about the quality of the data, and this can result in sophisticated analyses of bad data. But, as we all know, “Garbage in, garbage out.”

I blogged about this a couple of years ago in reference to Tertius Chandler’s list of city sizes through history, from both archaeological and historical sources (Link is here). Those data (Chandler 1987) are considered shockingly bad and worthless by most historical demographers and historians. In technical terms, they may be "bullshit" (see my post on bullshit).

Yet some urban scholars merrily use the data for studies today. I consider this a real problem, and said so in my review of a manuscript for a journal titled “Scientific Data” (my blog post was an elaboration on that review). But data quality issues were evidently not as important to the authors and journal editors; the paper was published with only a few weak caveats about the data (Reba et al. 2016).

Another recent case focuses on the identification of the plague and other diseases in historical sources. This one focuses on a database compiled in 1975 by Jean-Noël Biraben of historical occurrences of the plague in France and the Mediterranean area. Specialists recognize numerous biases and problems with the basic data. But once the data were digitized, non-historians readily used them without question, leading to problematic results. The basic problem was pointed out by Jones and Nevell (2016), and elaborated on by Roosen and Curtis (2018).

Plague or not plague? From Jones/Nevell)

I highly recommend Roosen and Curtis (and thanks to my excellent colleague Monica Green for sending this my way). Their remarks parallel my views of the Chandler city-size data, but they do a better job of articulating the historiographical issues involved when subsequent scholars used these data (badly):

“When scholars fail to apply source criticism or do not reflect on the content of the data they use, the reliability of their results becomes highly questionable.” (p. 103).

Jones and Nevell deplore “the loosening of the rigorous standards of evidence and interpretation scientific researcher typically demand within their own disciplines.” (103)

Roosen and Curtis list three problems with recent analyses of the Biraben data (including a paper in PNAS): “First, reflection on the data collection process has been improper; second, what the data represent has not been recognized; and third, critique of the original sources has been inadequate. We argue that a critical consideration of any of these three elements would have led to the conclusion that the data set should not have been used at face value.” (105)

“However, through digitization and subsequent publication in a top-ranked journal, the 4-decade-old data set was imbued with a false aura of trustworthiness and the impression of being new historical research.” (104)

Joosen and Curtis make some recommendations for improving the systematic analysis of historical disease data. First, scholars need to employ basic historiographical techniques when they use data. That is, scholars need to subject data to source criticism, comparison of sources, contextual analysis. Second, hypothesis testing should be done on limited regions, rather than Europe as a whole (because of regional biases in Biraben’s material). And third, scholars should compile new, better databases. According to Monica Green, this is now in process with a new international collaborative project.

I think I’ve run out of steam; I’ll have to discuss Binford’s hunter-gatherer database in a separate post. This is a complicated and very troubling case, and I’ve been delaying writing about it for some time now, so I guess I’ll wait a bit longer.

So, what about big-data archaeology. Do we have any bad data? “Not me!” we all exclaim. Nevertheless, we all know that some of our colleagues are sloppier than others, and that some data are more reliable than others. Here are two suggestions.

First, take a historiographic approach to data, both your own and those of others. Historiography refers to the basic methods that historians use to evaluate their sources. Compare sources. Analyze possible biases in the sources. Analyze the context, purpose, and situation of the process of generating or gathering the data. Good metadata standards is a start, but I think we need to go further. Check out some of the methodological literature in history: (Henige 2005), (Hansen and Hansen 2016), (Kipping et al. 2014), and an old favorite of mine: (Fischer 1970); I still kick myself for not taking a class with Fischer when I was at Brandeis!

Second, archaeologists should work out methods of categorizing the reliability of our data and findings, and these should become part of the metadata of any database. There are basic methods of assessing the reliability of data and findings (6 and Bellamy 2012; Gerring 2012; Hruschka et al. 2004). The Intergovernmental Panel on Climate Change has worked out a useful system for coding the reliability of findings by different authors. They suggest evaluating independently the strength of the evidence, and the agreement among authorities (Adler and Hirsch Hadorn 2014).

These suggestions will have to be adapted to use with archaeological data. But they can help us avoid some of the problems that have arisen with the use of the faulty databases on city size and plague occurrence described above. As archaeologist rush ahead into the brave new world of big data, we should try to fix data problems sooner rather than later. We should try to ensure that big data are not bad data.

6, Perri and Christine Bellamy

2012 Principles of Methodology: Research Design in Social Science. Sage, New York.

Adler, Carolina E. and Gertrude Hirsch Hadorn

2014 The IPCC and Treatment of Uncertainties: Topics and Sources of Dissensus. Wiley Interdisciplinary Reviews: Climate Change 5 (5): 663-676.

Chandler, Tertius

1987 Four Thousand Years of Urban Growth: An Historical Census. St. David's University Press, Lewiston, NY.

Fischer, David Hackett

1970 Historians' Fallacies: Toward a Logic of Historical Thought. Harper, New York.

Gerring, John

2012 Social Science Methodology: A Unified Framework. 2nd ed. Cambridge University Press, New York.

Hansen, Bradley A. and Mary Eschelbach Hansen

2016 The historian's craft and economics. Journal of Institutional Economics 12 (2): 349-370.

Henige, David P.

2005 Historical Evidence and Argument. University of Wisconsin Press, Madison.

Hruschka, Daniel J., Deborah Schwartz, Daphne Cobb St. John, Erin Picone-Decaro, Richard A. Jenkins, and James W. Carey

2004 Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods 16 (3): 307-331.

Jones, Lori and Richard Nevell

2016 Plagued by doubt and viral misinformation: the need for evidence-based use of historical disease images. The Lancet Infectious Diseases 16 (10): e235-e240.

Kipping, Matthias, R. Daniel  Wadhwani, and Marcelo Bucheli

2014 Analyzing and Interpreting Historical Sources: A Basic Methodology. In Organizations in Time: History, Theory, Methods, edited by Marcelo Bucheli and R. Daniel Wadhwani, pp. 305-330. Oxford University PRess, New York.

Reba, Meredith, Femke Reitsma, and Karen C. Seto

2016 Data Descriptor: Spatializing 6,000 years of global urbanization from 3700 BC to AD 2000. Scientific Data 3 (160034).

Roosen, Joris and Daniel R. Curtis

2018 Dangers of Noncritical Use of Historical Plague Data. Emerging Infectious Disease journal 24 (1): 103-110.  http://wwwnc.cdc.gov/eid/article/24/1/17-0477. 

Teotihuacan fracas: Pasztory claims she was ripped off and ignored by Millon & Cowgill

Esther Pasztory

I just read a strange and inflammatory paper by Esther Pasztory in the Mexican journal, Anales de Antropología (Pasztory 2017). Pasztory, a senior art historian and Teotihuacan scholar, raises questions about the scholarship and perhaps the ethics of two other top Teotihuacan scholars, René Millon (deceased) and George Cowgill. I have three main questions about this paper:

1. Did Millon really steal her ideas?  (the answer is, no).

2. Did Cowgill refuse to give her sufficient credit for her insights?  (the answer is, no).

3. Why would a reputable journal publish this paper? (the answer is, I have no idea).

 

Rene Millon

This paper focuses on the notion that the government and society of ancient Teotihuacan were more collective or corporate than most ancient societies. This view has been gaining in popularity recently. Pasztory claims to have invented the idea although the published record casts doubt on her claim. Pasztory’s 1987 book was a major early statement of this position. Cowgill (but not Millon) also contributed to the growing consensus that Teotihuacan may have had a more collective form of rule, using the term “oligarchy” and comparing Teotihuacan to Rennaissance Venice in several works. In recent years, Linda Manzanilla has been the major proponent of the collective rule position (Froese et al. 2014; Manzanilla 2002, 2006, 2015), and David Carballo has at least one paper in press arguing for collective rule at the site (see also Carballo 2016). This is not by any means a unanimous viewpoint; Saburo Sugiyama is the most vocal proponent of the single-king autocratic model for Teotihuacan (Sugiyama 2005, 2013; Sugiyama and Cabrera Castro 2007). Personally, I am on the fence. I think the collective model is probably valid, but I do not feel that the arguments in its favor have been particularly strong in empirical terms.

Pasztory has a variety of complaints about Millon and Cowgill. I will limit myself to the most egregious.

Complaint #1: Cowgill ignores her ideas and work.

 

George Cowgill

The impetus for this article was the publication of Cowgill’s recent synthesis of research on Teotihuacan (Cowgill 2015), which—she claims—ignores her and does not cite her enough times. She claims that the notion of collective rule was her idea in the first place: “esta idea fuse de mi autoría” (p. 219), or, “Soy la responsible del origen de algunas de las ideas centrales sobre la naturaleza de Teotihuacan” (p. 217). Cowgill discusses the collective rule idea, but does not cite Pasztory as its originator.

If one just reads Pasztory’s paper and looks at Cowgill’s book, one could come to divergent conclusions. First, only two of her works are cited in the bibliography, so perhaps she is correct about being ignored or slighted. But her six entries in the index are only exceeded by a greater number of entries for René Millon and Saburo Sugiyama. That is, Cowgill mentions her by name more than any other scholar except for these two. He clearly does take her seriously.

Let’s look at the history of George Cowgill’s ideas about collective rule at Teotihuacan. I will be brief here; however, it would be useful to write a more scholarly account in the future.

(Cowgill 1983) – In this discussion of government at Teotihuacan, focusing on the architectural compound known as the Ciudadela, Cowgill does not specifically use the terms collective or corporate rule. But he was clearly exploring the concept of non-autocratic rule, talking about how there may have been powerful individuals ruling the state instead of a single ruler. Instead of saying the Sun Pyramid was built by a single ruler, he says it was initiated by “persons.” He talks about the possibility of priest-rulers and uses the concept theocracy. He talks about rule by a variety of officials. To me, it looks like he was exploring the notion of collective rule, but without the benefit of an appropriate label or concept. I must admit that I never liked this paper; I thought the writing was wishy-washy, and at the time was convinced that rulership must have been despotic. But then we all thought like that back in the early 1980s.

1983-1991: Cowgill was publishing methodological papers and some quantitative studies of Teotihuacan during these years. He did not give much attention to Teotihuacan government.

(Cabrera Castro et al. 1991) – Cowgill and his coauthors talk about “a shift to more emphasis on a collective, group-oriented ethos” (p. 89) for Teotihuacan society and government.

(Cowgill 1992) – He cites Pasztory as suggesting a more corporate orientation for Teotihuacan rule. He quotes Millon (Millon 1981) as acknowledging that Teotihuacan rule might have been either “individual or collective” (Cowgill 1992:98).

(Cowgill 1997) – This is a major scholarly review article on “society and state” at Teotihuacan. He includes significant discussion of possibilities for collective rule, citing seven works by Pasztory. This is the place where Cowgill gives Pasztory’s views their most detailed consideration. He cites her newly-published book, which contains the fullest exposition of her views (Pasztory 1997). Although Pasztory does not cite this paper in her recent article, she might not consider the level of coverage of her ideas sufficient because—as in his book—Cowgill does not state explicitly that collective rule was her idea in the first place (Pasztory 2017:219). But, if my interpretation of Cowgill’s 1983 paper is correct, then he was exploring these concepts himself before she put her ideas into print.

So, why did Cowgill give Pasztory’s work such short shrift in his 2015 book? At least half of his mentions are critiques of her ideas about various notions, including the Great Goddess concept. I would suggest that he had explored her ideas previously and found them inadequate for his understanding of Teotihuacan society. I have a similar view, which I will describe below.

Complaint #2: Millon stole her ideas

According to Pasztory (p. 218), Millon read the draft of a chapter, subsequently published as (Pasztory 1988). She does not provide a date, but 1986 or 1987 would sound appropriate. The chapter described her ideas about collective rule, but Millon is said to have expressed vehement opposition to this concept (“se opuso vehementemente a la idea”, p. 218). Then, at the Dumbarton Oaks conference on Teotihuacan in 1988, she was surprised to hear Millon talk about collective rule at Teotihacan, while failing to give her any credit for the idea! (“sin darme ningún crédito por la idea”). Now these events are difficult to reconstruct today without a lot of interviews and piecing together the story. But I don’t think stories like this have much importance. What is important is the published record. What did Millon say in the published version of his paper and in other publications?

(Millon 1981) – I first go back in time to an earlier paper, not cited by Pasztory. In this review article, Millon states in passing that rulership at Teotihuacan could have been either “individual or collective” (p. 212). Millon was clearly thinking about this issue long before reading Pasztory’s unpublished chapter. But he does not develop the idea in this paper. As mentioned above, Cowgill (1992) later cited Millon (1981) as mentioning collective rule.

(Millon 1992) – This paper from the Dumbarton Oaks volume is a lengthy and detailed analysis of several decades of research at Teotihuacan. It is the published version of the conference talk in which Millon reportedly discussed Pasztory’s ideas without citation or credit. Millon devotes five pages (pp. 371-375) to the ideas of Pasztory about Teotihuacan society and rule! He cites six of her publications! This is hardly ignoring her, and far from stealing her ideas. He organizes her ideas into four main claims, analyzes them, and concludes that one claim survives the evidence, two are contradicted by evidence, and one survives, but is better explained in a different way. My interpretation of Millon’s complaints about Pasztory’s work (which I report from my 1990’s annotations in the margin of the article) is that it is insufficiently anthropological and too subjective. (One other relevant factor: As I know from recent experience, speakers are not given a great amount of time at Dumbarton Oaks conferences, and the schedule is followed tightly. One simply does not have time for a lot of scholarly citations in these oral presentations.)

So, did Millon steal the ideas of Pasztory? Hardly. In the 1992 chapter, he discussed the collective rule idea, engaging closely with her publications. But he is not convinced. Sadly, this was René Millon’s final major paper on Teotihuacan. He never articulated an integrated vision of government at Teotihuacan.

What are the main issues here?

(1) Humanities vs. social scientific scholarship. Pasztory seems to think that if she came up with an idea first, anyone who later engages with that idea must cite her as having originated it. Apart from the idea of whether she was indeed the first to propose a collective model for Teotihuacan, her expectations are out of step with the standard model of research and citation in the social sciences. When I write about the Aztec empire as being an indirectly control empire, I don’t feel the need to go back and cite the first person who may have proposed that idea (Ross Hassig, although he used the term “hegemonic empire.”) If I am writing a history of scholarship on the topic, of course I’ll credit Hassig. But if I am just going about my scholarly business today, I don’t need to invoke his name every time I talk about the organization of the empire. Perhaps in the humanities, the person who first articulates a concept needs to be acknowledged all the time. But, in the social sciences, the crucial issue is empirical: What do the data show? Both Millon and Cowgill examined Pasztory’s ideas and evidence carefully, and concluded that they were not needed in order to make their arguments. We lack later papers by Millon, but Cowgill’s trajectory is clear: he talks about her ideas in a 1997 review article, and then later references them only where he feels the need.

(2) Professional pride. Pasztory clearly feels that Millon and Cowgill insulted her professional pride by not giving her ideas enough consideration or citation. But, I think my chronological discussion above show that neither scholar was remiss in discussing her ideas and works. Should Cowgill have cited her more extensively in his book? He doesn’t really deal much with the history of interpretations of Teotihuacan, so I don’t fault him there.

(3) Collective rule at Teotihuacan. I am an agnostic when it comes to the collective rule interpretation of Teotihuacan. It has become something of a bandwagon. As a curmudgeon, I have a strong dislike for bandwagons. I prefer to sit back, consider the evidence, and write critiques of poorly supported popular notions. When I first moved back to Teotihuacan scholarship a couple of years ago, I eagerly went through Pastory’s main book (Pasztory 1997). Given my documented infatuation with Blanton and Fargher’s model (Blanton and Fargher 2008), I wanted to review and synthesize the evidence for collective rule at Teotihuacan. But I was disappointed. The book has some insights, but to me, most of her evidence is subjective and open to multiple interpretations. I didn’t find much that I consider rigorous empirical evidence to support a collective model for Teotihuacan government. I do cite her work, though, because her book remains an important work. I feel that archaeologists have yet to develop sufficiently rigorous methods to identify collective vs. autocratic rule with confidence, using archaeological evidence. I have read works by Manzanilla, Carballo, Blanton, Fargher, Feinman, and others on this issue, but I remain unconvinced. My gut feeling is that the collective model fits Teotihuacan better than the autocratic model. But I have not seen a sufficiently rigorous study, with enough evidence to convince me. (Yes, I know these people are probably frustrated at my curmudgeonly approach here. My mantra is, "Show me the data!").

(4) Why did Anales de Antropología publish this piece? It looks like no one checked Pasztory’s accusations against the published record. It would be useful if someone were to write a history of ideas about Teotihuacan government and society. But unfortunately, much of this paper sounds petty and unprofessional. It is published in a peer-reviewed journal, but was this paper subject to outside review? I have no idea.

REFERENCES:

Blanton, Richard E. and Lane F. Fargher

2008 Collective Action in the Formation of Pre-Modern States. Springer, New York.

Cabrera Castro, Rubén, Saburo Sugiyama, and George L. Cowgill

1991 The Templo de Quetzalcoatl Project at Teotihuacan: A Preliminary Report. Ancient Mesoamerica 2: 77-92.

Carballo, David M.

2016 Urbanization and Religion in Ancient Central Mexico. Oxford University Press, New York.

Cowgill, George L.

1983 Rulership and the Ciudadela: Political Inferences from Teotihuacan Architecture. In Civilization in the Americas: Essays in Honor of Gordon R. Willey, edited by Richard M. Leventhal and Alan L. Kolata, pp. 313-344. University of New Mexico Press, Albuquerque.

1992 Toward a Political History of Teotihuacan. In Ideology and Pre-Columbian Civilizations, edited by Arthur A. Demarest and Geoffrey W. Conrad, pp. 87-114. School of American Research Press, Santa Fe.

1997 State and Society at Teotihuacan, Mexico. Annual Review of Anthropology 26: 129-161.

2015 Ancient Teotihuacan: Early Urbanism in Central Mexico. Cambridge University Press, New York.

Froese, Tom, Carlos Gershenson, and Linda R. Manzanilla

2014 Can Government Be Self-Organized? A Mathematical Model of the Collective Social Organization of Ancient Teotihuacan, Central Mexico. PloS one 9 (10): e109966.

Manzanilla, Linda R.

2002 Gobierno corporativo en Teotihuacan: una revisión del concepto "palacio" aplicado a la gran urbe prehispánica. Anales de Antropología 35: 157-190.

2006 Estados corporativos arcaicos: organizaciones de excepción en escenarios excluyentes. Cuicuilco 13 (36): 13-45.

2015 Cooperation and tensions in multiethnic corporate societies using Teotihuacan, Central Mexico, as a case study. Proceedings of the National Academy of Sciences 112: 9210-9215.

Millon, René

1981 Teotihuacan: City, State, and Civilization. In Archaeology, edited by Jeremy Sabloff, pp. 198-243. Supplement to the Handbook of Middle American Indians, vol. 1. University of Texas Press, Austin.

1992 Teotihuacan Studies: From 1950 to 1990 and Beyond. In Art, Ideology, and the City of Teotihuacan, edited by Janet C. Berlo, pp. 339-429. Dumbarton Oaks, Washington, DC.

Pasztory, Esther

1988 A Reinterpretation of Teotihuacan and its Mural Painting Tradition. In Feathered Serpents and Flowering Trees: Reconstructing the Murals of Teotihuacan, edited by Kathleen Berrin, pp. 45-77. Fine Arts Museums, San Francisco.

1997 Teotihuacan: An Experiment in Living. University of Oklahoma Press, Norman.

2017 Nota: Panorama de los estudio sobre Teotihuacan: un corrección historiográfica: Adding dimension to studies on Teotihuacan: A historiogrpahic corection. Anales de Antropología 51: 217-221.

Sugiyama, Saburo

2005 Human Sacrifice, Militarism, and Rulership: The Symbolism of the Feathered Serpent Pyramid at Teotihuacan, Mexico. Cambridge University Press, New York.

2013 Creation and Transformation of Monuments in the Ancient Citiy of Teotihuacan. In Constructing, Deconstructing, and Reconstructing Social Identity: 2,000 Years of Monumentality in Teotihuacan and Cholula, Mexico, edited by Saburo Sugiyama, Shigeru Kabata, Tomoko Taniguchi, and Etsuko Niwa, pp. 1-10. Aichi Prefectural University, Cultural Symbiosis Research Institute, Aichi.

Sugiyama, Saburo and Rubén Cabrera Castro

2007 The Moon Pyramid Project and the Teotihuacan State Polity. Ancient Mesoamerica 18: 109-125.

The speculation scale (the inverse of empirical adequacy)

ASU student Lisa Gallagher in our lab

I am posting this from the ASU lab at Teotihuacan in Mexico. I will be attending a conference on Teo sponsored by UNAM and Penn State over the next few days. On the trip down I read one of the worst articles I've read in a long time. I was surprised that the paper was accepted by a journal. It was published in a new journal, Economic Anthropology, whose standards have perhaps not risen to a level the editors would like. Did anyone review this paper?

This bad article got me thinking about the ratio between the scope or breadth of the claims made in a study, and the amount of data used. Works with a low ratio are often called "descriptive" studies. Works with a high ratio, on the other hand, contain little data, but make sweeping claims that go far beyond the data at hand. These are speculative studies, work that is poorly grounded. The paper I just read had a very high ratio, which is why I disliked it so much. Satisfying research in archaeology (and the social and historical sciences generally) usually falls in the mid-range of this scale.

I decided to create a graphic to illustrate this speculation scale. First I created nine data-to-interpretation schemes. These are combinations of three categories: low medium, and high amounts of data (blue circles), and low, medium and high levels of interpretation (red squares). I label these schemes from one  to nine, and arrange them on the speculation scale (see graphic).

Scheme 1 occupies the low, or descriptive, end of the scale. Those enamored of theory often dismiss such studies as "merely descriptive." But those of us who like to analyze data often find these studies useful as a source of data (that is, IF the data are adequately described and derive from rigorous methods, etc.). Schemes 2 and 3 are also descriptive in nature: the amount of data exceeds the amount of interpretation.

At the top of the scale, scheme 9 pertains to what I consider to be useless works. Who needs a bunch of speculation with little or no data? Unless the work is an editorial or opinion essay, or a work of pure theory, papers illustrating scheme 9 that purport to be empirical advances in fact contribute little to scholarship. How do these things get past reviewers and editors? But they do. Schemes 7 and 8 are also speculative in nature, although less flagrant than scheme 9. Quite a few studies in archaeology fit here. Most work in postprocessual and "social" archaeology probably correspond to schemes 7 and 8.

The schemes that fall in the middle of the speculation scale--schemes 4, 5, and 6--are those that seem most satisfying to most archaeologists. They are empirical studies based on real data that employ concepts and theories for explanation or interpretation. The amount of data is balanced by the amount of interpretation; the level of interpretation matches the amount of data.

*** ADDITION, Jan 30:  This graphic is probably a simpler way of showing the relationship:

*********

Obviously, this is a simplistic device for looking at the quality of publications. It says little about the quality of the data, or about the fit between interpretation and data. If you apply inappropriate models, your results will be suspect, even if the data and models are both of good quality on their own. But schemes like this can help us think about the adequacy of our models, interpretations, and explanations. If you have only a limited amount of data, then you should probably bypass big elaborate explanations and try to find something on a smaller scale. I know the postprocessualists and postmodernists will think I am being regressive here, but if archaeology is to be a science and generate reliable knowledge about the past, then we need to be able to match up our data with our interpretations/explanations. For more along these lines, see my series on archaeology as a science, starting with this post, or see my forthcoming paper:

Smith, Michael E.

2017?   Social Science and Archaeological Inquiry. Antiquity  (in press).

The field of economic anthropology has generally been relatively empirical in orientation, with good epistemology. I would think that a journal with that title would know better than to publish a paper whose score on the speculation scale (scheme 9) is so high. Ugh.

Have we gotten out of the crisis in Latin American book reviews?

I have always been a big fan of book reviews. When I get a new journal, I may scan the article titles first, but I almost always read the book reviews before the articles. Book reviews are an important part of quality control in scholarly disciplines where books are prominent (as in archaeology). In the past I have blogged about the book review crisis in Latin American archaeology: See my posts in:

2008,  2009,  2011

For a decade, very few books were being reviewed in the main journal, Latin American Antiquity. The major Mesoamerican journal, Ancient Mesoamerica, doesn't published book reviews at all! But now, book reviews are trending up in quantity. I thought there were more reviews in LAA over the past year, so I counted them up. Here are the data, starting in 1997.

These figures paint a bleak picture of the decline of quality control in Latin American archaeology starting around 2004. But after six years of almost no reviews (average of one--count, 'em--1 per issue!), things are trending up in 2016.

Book reviews are important for many reasons. First, they get out the news about new publications. Second, a good book review is a gem of a short essay on the topic of the book. Because I try to keep up with urban research at some level in many different disciplines, I use book reviews to help guide my reading. And third, book reviews give an indication of the quality of the book. Bad books are called out, and good books are praised. I recently had to give a quick judgment on the quality and influence of a book in a field far from my own (NOT Latin American archaeology!). I was able to find four book reviews easily, and they gave me the information I needed.

Let's hope this trend at Latin American Antiquity continues! Perhaps Ancient Mesoamerican might be persuaded to begin reviewing books. When I have suggested this at the board meetings, the response has been, "Fine, if you want to organize it, go ahead," hardly an enthusiastic promise of support.

If you are asked to review a book, please do it. If you would like to see more book reviews, contact the relevant journal editors and let them know. Our field has need of all the quality control we can get, and book reviews should be a major part of our collective strategy of disciplinary improvement.

Why would a journal called "Scientific Data" publish bad data? The Chandler/Modelski city-size problem

Scholars interested in changes in city size over long periods of time often turn to one or both of two encyclopedia compilations of data: Tertius Chandler’s, Four Thousand Year of Urban Browth (Chandler 1987), and George Modelski’s World Cities: -3000 to 2000 (Modelski 2003). Chandler’s book is an update of an earlier version (Chandler and Fox 1978). The data in both Chandler and Modelski are a mess, routinely  dismissed by urban demographic historians as worthless for serious scholarship. Yet a growing number of scholars—particularly economic historians—mine those sources for city-size data in order to investigate various questions. This situation brings up a number of thorny professional and ethical questions.

In this post I describe the situation and point out some of the troubling questions that come to mind. This is not a thorough exploration of either realm. I only have two more days at the lab in Teotihuacan, Mexico, and I have about four days worth of tasks to complete.

I am moved to write this because of a new paper relying on Chandler and Modelski’s bad data was just published in a new journal called Scientific Data (Reba et al. 2016).

What is wrong with the data of Chandler and Modelski?

The new paper (Reba et al. 2016) describes these sources. How they were compiled, and various problems and difficulties with the data. The latter are mostly limited to context, measurement, and presentation issues. The basic question of whether the data are accurate is barely considered. I am not a demographic historian, so I cannot provide a detailed critique. But I do use systematic archaeological and historical data on city size in some of my research, so I have looked into this question. I will limit myself to quoting experts (that is, urban demographic historians, scholars with experience working with primary sources). Some of these refer to the earlier version of Chandler (Chandler and Fox 1978), but the methods and reliability are not very different between the two editions.

(de Vries 1984)

·         “Three Thousand Years of Urban Growth by Tertius Chandler and Gerald Fox, is a massive collection of information about the size of cities. Its unsystematic character, and, even worse, the authors’ reliance on suspect sources and their completely uncritical use of such sources renders the volume all but unusable.” (p. 18)

(Hopkins 1978)

·         “the compendious, useful but not obviously reliable T. Chandler and G. Fox,” (p. 2)

(Bairoch 1988)

·         Chandler and Fox:  “But it is only after the year 800 that this survey may be regarded as truly systematic. This monumental and extremely useful work nevertheless leaves a number of important gaps unfilled. These gaps result from the omission of a fairly sizeable fraction _ / - of the cities of the world (probably 20% of the larger cities and 60% of the smaller ones) and especially from the failure to undertake a systematic review of recent studies on the history of various individual cities.”  (p. 116-117)

(Rozman 1978)

·         “Chandler and Fox fail to use sources in languages essential for their task” (p.66)

(Binford 1975)

·         There are “serious flaws in the work” (p.23)

·         [Chandler and Fox employ] “dubious rules of thumb”

·         “Statements and methods like these undermine confidence in the work as a whole, because they suggest a fundamental naiveté … The authors overestimate the importance of statistics in the thinking of our ancestors, and assume too much uniformity in pre-industrial population trends. For all the cautious skepticism proclaimed in their introduction, they still seem too willing to make census taker out of medieval travelers, and to assume that, in the absence of factories and international capitalism, one place was more or less like another. This is not a work informed by the research of other scholars on the various reasons for collecting data in the past, or in the complexity of population trends even in small, non-industrial cities. Without more attention to such matters, the reader can only guess at the margins of error he ought to attach to the Chandler-Fox statistics, and therefore cannot use them with assurance in further calculations (for example, in computing rates or indices with gross population as a base.” (p. 23-24)

·         “One of the weaknesses of the Chandler-Fox collection is that the authors are sometimes too gullible in dealing with their informants. Similar errors may be introduced by mis-enumeration on the part of those who first gathered the evidence.” (26-27)

(Kowalewski 1990)

·         “Issues of completeness, scale, and time-depth make some studies (e.g.. Chandler 1987) less appropriate for comparative purposes.” (p.40)

(Woods 2003)

·         Critical of Chandler, but uses the data

·         Chandler’s data are  “more doubtful, at least for some regions.” (p. 219)

(Chase-Dunn et al. 2005:97)

·         Chandler's “estimates are obviously error-prone.”

Personal communications

In order to make sure my view of this topic was up-to-date, I emailed several urban demographic historians (about a year ago). I did not ask for their permission to quote their remarks, so I include the following quotations as anonymous. These two are well-respected historians, each of whom has published books and articles on population reconstruction in the past.

Scholar 1:

·         “Many years ago I thought about writing a paper on this/these books (There are two editions, I think) but never did get around to writing it. But as far as I'm concerned it's all but worthless.  Chandler was assiduous, without a doubt, but just compiled (or piled) one estimate after another without any indications of the merits or demerits of each. It might be useful for tracking down estimates but not for the estimates themselves.

·         Modelski  I don't know--yet.”

Scholar 2:

·         “I do not have a high opinion of Chandler’s data. In those cases where I was able to check his Greek, Roman, medieval or early-modern data, they turn out to be seriously wrong. Do not know about the more modern data, though.”

A note on these scholars

Most historians are notoriously picky about their sources; they like to stick close to their textual sources and hesitate to compare or generalize beyond one or two cases (Grew 1990; Kocka 2003). Thus one might be tempted to reject the above critiques as reflecting this bias. Some historians simply do not accept the validity of broad comparative analyses that require simplification and standardization of diverse local datasets. I discuss this and related issues of comparative analysis elsewhere (Smith and Peregrine 2012). But most or all of the individual quoted above actively pursue comparative analysis. They have all assembled regional or temporal databases of city-size data, although not on the scale of Chander and Modelski. While they are well aware of the historiographic issues of data sources, they are also willing and able to systematize, simpilify, and push forward with comparative data. In other words, their critiques do not reflect the cranky complaints of particularistic historians who are anti-comparativist. Instead, their critiques reflect real historiographical issues in the origins, quality, and relevance of primary data on city sizes in the past.

Some positive comments

(Binford 1975)

·         “This book will be useful to students of population and urban development because it is the only worldwide compilation done by one person using consistent criteria.” (p. 22)

Christopher Chase-Dunn and Daniel Pasciuti reviewed Modelski very positively in Jr. World Systems Research, 2004. They work with Modelski and have co-published with him. They evidently liked the book so much that they later published the IDENTICAL book review, word-for-word, in the journal Globalizations, in 2006.  Hmmmmmmm……..

Uses of these data by scholars seemingly oblivious to potential historiographical problems:

·         (Reba et al. 2016)

·         (Manning 2005)

·         (Jedwab and Vollrach 2015)

·         (Jedwab and Vollrach 2016)

·         (Nunn and Qian 2011)

Uses of the data by scholars who attempt to verify or adjust the data in relation to other scholarship

(Morris 2013:146)

·         Ian Morris uses some of the data cautiously, in the context of a discussion of it’s validity:

·         “in my opinion some of Chandler and Fox’s estimate are not supported well by the data.”

·         “While there would be some advantages to taking a single source like Chandler and Fox’s Three Thousand Years of Urban Growth and then relying on it consistently, the drawbacks seem to outweigh them.” (p. 146)

(Acemoglu et al. 2002)

·         These scholars analyze Chandler’s data, compare them to, Bairoch’s data, and attempt to evaluate their usefulness.

Troubling questions

My gut reaction to works like the new paper by Reba et al, is “Garbage in, garbage out.” The data cannot be trusted, so why should we be expected to trust the results of the analysis? Here are some troubling questions that arise out of this situation.

(1) Why do otherwise rigorous scholars feel free to use bad data?

·         Because it is there. If data exist in some usable format, and they seem relevant to a research question, someone will use the data, even if the data are terrible, unreliable, or inaccurate. I have considered posting a bogus dataset online to see if people will analyze it. The risks seem to outweigh the benefits, though.

·         Because scholars lack training in other disciplines. The paper by Reba et al is curious in that it has a very rigorous discussion of geospatial methods, but virtually no discussion of the historiography of the city size data. See some of the sources quoted above for historiographical discussions of historical city-size data.

·         Because scholars are not critical of sources. Perhaps there is an assumption that all data in other disciplines are valid. Perhaps scholars believe that if something is published it is true. Do the authors think that data published by non-experts in another discipline (neither Chandler nor Modelski are/were urban historians or demographic historians) are valid just because they are published? Reba et al justify their use of Chandler’s data based on the fact it is published and other scholars have used it. They provide citation data for Chandler and Modelski. The message is that if it is published and cited, it must be valid. Hmmm, I haven’t encountered that principle in my readings on methodology in the social sciences or the historical sciences.

(2) Why don’t scholars who know the data well care about this?

        Scholars like Bairoch, de Vries, and others quote above DO care about data quality, and hence their negative remarks on Chandler and Modelski’s data. But many of these quotations come before the time when economic historians and others started mining data sources for city size data. Perhaps these and other historians simply don’t care that historical data are being used badly. With the advent of the Internet, there are all sorts of bad data readily available, and all sorts of bad analyses of bad (and good) data. Or perhaps disciplinary myopia is the cause. Urban historians are not reading journals like Scientific Data, Explorations in Economic History, so perhaps they are unaware of the uses of Chandler and Modelski’s data.  Perhaps they have trouble believing that such terrible data would be taken seriously by scholars.

This latter is a very real factor in work that crosses disciplinary boundaries. For years I didn’t believe that Jane Jacobs’s silly and ridiculously inaccurate model that cities preceded agriculture could be taken seriously by anyone at all. Then I found that geographers were citing and praising the model. I got cranky and did a blog post or two on this. Then when a paper in a major journal promoted this idea, I finally got motivated to publish a critique, so I rounded up a few colleagues and we published a paper (Smith et al. 2014). Maybe the urban historians have not yet been provoked sufficiently to mount a proper attack on Chander and Modelski.

(3) What can be done about this situation?

I have to admit that I really despair of this situation. I am very upset that such obviously poor data are being used by otherwise rigorous scholars, and I am upset that I don’t have better data. I have talked to quite a few colleagues—archaeologists and ancient historians—about this situation. I have asked if any of them were involved in assembling reliable and accurate data on ancient city sizes in their region of specialty, and the answer has been negative. I have asked if they knew of anyone doing systematic urban demographic history in their region, and again the answer is no. In my own region, Mesoamerica, there was a flurry of demographic work on city size in the 1980s, but then scholars lost interest. I have asked if anyone might be interested in mounting such a systematic comparative project, again with a negative answer. This may not arise entirely from a lack of interest – if someone asked me to assemble demographic data from all Mesoamerican cities at, say, 50- or 100-year intervals, I might claim that I am too busy for such a task. It would take a lot of time, and it is hard to see how a granting agency would get excited about such a project.

I don’t have any grand conclusions here. The situation is bad--journals and scholars are merrily using bad data--and I don’t have any good solutions, beyond the suggestion that bad data should be avoided. The journal Scientific Data should be ashamed for using such non-scientific data. We can all do better than this.

References

Acemoglu, Daron, Simon Johnson, and James Robinson

2002  Reversal of Fortune: Geography and Institutions in the Making of the Modern World. Quarterly Journal of Economics 117 (4): 1231-1294.

Bairoch, Paul

1988  Cities and Economic Development: From the Dawn of History to the Present. Translated by Christopher Braider. University of Chicago Press, Chicago.

Binford, Henry C.

1975  Never Trust the Census Taker, even when he'd Dead. Urban History 2: 22-28.

Chandler, Tertius

1987  Four Thousand Years of Urban Growth: An Historical Census. St. David's University Press, Lewiston, NY.

Chandler, Tertius and Gerald Fox

1978  Three Thousand Years of Urban Growth: An Historical Census. Academic Press, New York.

Chase-Dunn, Christopher, Alexis Álvarez, and Daniel Pasciuti

2005  Power and Size: Urbanization and Empire Formation in World-Systems Since the Bronze Age. In The Historical Evolution of World-Systems, edited by Christopher Chase-Dunn and E. N. Anderson, pp. 92-112. Palgrave Macmillan, London.

de Vries, Jan

1984  European Urbanization 1500-1800. Harvard University PRess, Cambridge.

Grew, Raymond

1990  On the Current State of Comparative Studies. In Marc Bloch aujourd'hui: Histoire comparée y et sciences sociales, pp. 323-334. Éditions de l'École de Hautes Études en Sciences Sociales, Paris.

Hopkins, Keith

1978  Conquerors and Slaves. Cambridge University Press, New York.

Jedwab, Remi and Dietrich Vollrach

2015  Urbanization without Growth in Historical Perspective. Explorations in Economic History (under review??) 58: 1-21.

2016  The Urban Mortality Transition and the Rise of Poor Mega-Citiesunpublished paper posted online.

Kocka, Jürgen

2003  Comparison and Beyond. History and Theory 42: 39-44.

Kowalewski, Stephen A.

1990  The Evolution of Complexity in the Valley of Oaxaca. Annual Review of Anthropology 19: 39-58.

Manning, Patrick

2005  Migration in World History. Routledge, New York.

Modelski, George

2003  World Cities: -3000 to 2000. Faros 2000, Washington, DC.

Morris, Ian

2013  The Measure of Civilization: How Social Development Decides the Fate of Nations. Princeton University Press, Princeton.

Nunn, Nathan and Nancy Qian

2011  The potato's contribution to population and urbanization: Evidence from an historical experiment. Quarterly Journal of Economics 126 (2): 593-650.

Reba, Meredith, Femke Reitsma, and Karen C. Seto

2016  Data Descriptor: Spatializing 6,000 years of global urbanization from 3700 BC to AD 2000. Scientific Data 3 (160034).

Rozman, Gilbert

1978  Urban Networks and Historical Stages. Journal of Interdisciplinary History 9 (1): 65-91.

Smith, Michael E. and Peter Peregrine

2012  Approaches to Comparative Analysis in Archaeology. In The Comparative Archaeology of Complex Societies, edited by Michael E. Smith, pp. 4-20. Cambridge University Press, New York.

Smith, Michael E., Jason Ur, and Gary M. Feinman

2014  Jane Jacobs’s 'Cities-First' Model and Archaeological Reality. International Journal of Urban and Regional Research 38 (4): 1525-1535.

Woods, Robert

2003  Urbanisation in Europe and China during the second millennium: a review of urbanism and demography. International Journal of Population Geography 9 (3): 215-227.