Graphiti

by Jonathan Robinson on September 20, 2011 · 9 comments

in Data,Graphiti,War

In honor of the 10th anniversary of 9/11, Wired Magazine visualizes the uncertainty around the true number of Iraqi casualties since the US military engagement began in 2003. While the US military has accurate, continuously updated numbers on casualties amongst its servicemen and women, there is considerable disagreement about the level of Iraqi casualties.  Wired has provided all the data and their sources for these and related graphs for its readers.

{ 9 comments }

Mark Rogers September 20, 2011 at 5:49 pm

The red dot representing US servicemen is way to big if it is supposed to correspond to the rest of the graph.

Carlisle Rainey September 20, 2011 at 7:43 pm

This graph illustrates the differences between infographics and statistical graphics. This graph is intended to catch the reader’s eye and pique curiosity. It does that well I think. Like Mark, I am not sure how to interpret the size of the circles, so I made a less flashy, but more informative graph of my own. http://wp.me/p1RDVT-4H

Michael Spagat September 21, 2011 at 6:59 am

Jonathan Robinson says that Wired provides methodologies for the various numbers they report but I don’t see anything there about methodologies. Am I missing something?

In fact, it seem to me that the main problem with the graphic really is that it treats all numbers as equally credible, no matter how badly founded some of them are.

The graphic ignores a large literature on these numbers. On the Burnham et al. study (the Lancet report), just confining myself to peer-reviewed publications I could cite:

1. Wartime estimates of Iraqi civilian casualties, by Beth Osborne Daponte. International Review of the Red Cross, No. 868. http://www.icrc.org/eng/resources/documents/article/review/review-868-p943.htm

2. Bias in Epidemiological Studies of Conflict Mortality, by Neil F. Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, Gesine Reinert. Journal of Peace Research, Vol. 45, No. 5. http://jpr.sagepub.com/cgi/content/abstract/45/5/653

This won the article of the year award at the Journal of Peace Research:

http://www.prio.no/Research-and-Publications/Journal-of-Peace-Research/Article-of-the-year/Article-of-the-Year-2008/

3. Sampling bias due to structural heterogeneity and limited internal diffusion, by Jukka-Pekka Onnela, Neil F. Johnson, Sean Gourley, Gesine Reinert, Michael Spagat. http://iopscience.iop.org/0295-5075/85/2/28001/pdf/epl_85_2_28001.pdf

4. Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq, by Michael Spagat, Defense and Peace Economics, http://www.informaworld.com/smpp/content~db=all~content=a921401057

The journal invited Burnham et al. to respond to this paper but they did not rise to the occasion.

5. Retrospective two-stage cluster sampling for mortality in Iraq, by Seppo Laaksonen, International Journal of Market Research, Vol. 50, No. 3, 2008. http://mathstat.helsinki.fi/msm/banocoss/abst/Laaksonen.pdf (Unfortunately the article itself is gated)

6. Confidence Intervals for the Population Mean Tailored to Small Sample Sizes, with Applications to Survey Sampling, by Michael Rosenblum, Mark J. van der Laan. University of California, The International Journal of Biostatistics, vol. 5. issue 1, 2009. http://www.bepress.com/ijb/vol5/iss1/4/

7. Mainstreaming an Outlier: The Quest to Corroborate the Second Lancet Survey of Mortality in Iraq , Defense and Peace Economics. http://personal.rhul.ac.uk/uhte/014/Mainstreaming.pdf

Again a response was requested from Burnham et al. but they did not provide one.

If we expand only to substantial reviews rendered by peers that have not appeared in peer-reviewed journals then there is much more, e.g.,

8. A rare formal censure of Gilbert Burnham by the American Association for Public Opinion Research:

http://www.aapor.org/Content/NavigationMenu/PressRoom/RecentPressReleases/AAPORFindsGilbertBurnhaminViolationofEthicsCode/default.htm

and

http://www.aapor.org/uploads/AAPOR_Press_Releases/BurhnamDetailWebsite.pdf

9. Gilbert Burnham was suspended by Johns Hopkins for ethics violations despite what seems to have been quite a half-hearted investigation:

http://www.jhsph.edu/publichealthnews/press_releases/2009/iraq_review.html

On the ORB number see:

Conflict Deaths in Iraq: A Methodological Critique of the ORB Survey Estimate with Josh Dougherty, Survey Research Methods, Vol 4, No 1, 2010, pp. 3-15.

http://w4.ub.uni-konstanz.de/srm/article/view/2373

and a debate about this paper with Johnny Heald of ORB:

Discussion of Conflict Deaths in Iraq: A Methodological Critique of the ORB Survey Estimate with Josh Dougherty and Johnny Heald, Survey Research Methods, Vol 4, No 1, 2001, pp. 17-19.

http://w4.ub.uni-konstanz.de/srm/article/view/4269

There has been much less discussion of the ORB number than the Burnham et al. one because very few people ever took the ORB one seriously in the first place.

Obviously, the graphic would look very different with ORB and Burnham et al. gone. However, it still would be a very poor graphic for a number of reasons including that the numbers refer to widely varying time frames.

Mike Spagat

mikeprice42 September 21, 2011 at 10:19 am

Agree with the points raised by Mr. Spagat above. The ORB number has been shown to be just baseless and wrong. I’d say the same is clearly true of the Lancet survey (as per all the links Spagat provides above, and more) but there’d probably still be somewhat more debate about that than ORB.

Another issue with the graph in addition to those already mentioned is the problem of death classification. The notes in the graph say these are civilian or all violent deathse, depending on what was given, but that is a big difference, and is not a difference of ‘uncertainty’ which the graph claims to be illustrating. I suspect this will be quite confusing to many readers too. Note that Rainey’s graph (comment 2) refers to his version as “Estimated Number of Civilian Deaths”, when most of the sources aren’t that.

And why is only the ‘Civilian’ number (66,081) used for the Wikileaks war log number when it provides a number for all Iraqi violent deaths (civilian and combatant, as most of the other sources are doing)?

Carlisle Rainey September 21, 2011 at 1:14 pm

Thanks for the tip. I changed the title of my preferred version to be more vague, thus more accurate. When there is a tradeoff between vague and accurate variable descriptions, the data probably don’t belong on the same plot. My mistake illustrates why.

Adano September 21, 2011 at 11:16 am

Carlisle: Thanks for turning this into a legible graph.

Rob M. September 21, 2011 at 12:08 pm

This new Graphiti feature has been pretty disappointing so far, e.g. this post, O’Reilly vs. Stewart, grading schools. The above graph is the type of example I would use in an intro stats course to show students what not to do.

I realize Graphiti is supposed to be a fun, casual feature, but why can’t the the graphs be interesting and also portray quantitative info accurately? It’s frustrating to to see this stuff an academic blog whose authors are expected to have a decent, if not advanced, grasp of data visualization.

John Sides September 21, 2011 at 12:21 pm

Rob M.: Graphiti is not a showcase for the best in statistical graphics. Some graphs will be teh awesome; others will be flawed. As I’ve said before, the goal is just to pick something interesting — i.e., to stimulate interest. Judging by the comment thread, in which Carlisle provided a new version of the graphic and other commenters extensive detail about the studies it presents, I would say that this post accomplished that goal.

If you still don’t like the Graphiti posts thus far, feel free to send us suggestions for future posts.

Mark Rogers September 21, 2011 at 1:56 pm

I find the graphiti post fascinating.

Comments on this entry are closed.

Previous post:

Next post: