Gary King writes:

This article in the Washington Post quotes an intelligence official that “The chance of a false positive from the DNA testing [that it was bin Laden] is “approximately one in 11.8 quadrillion.” I think the people in this field haven’t the slightest idea what 11 quadrillion really means or what the claim of a ‘false positive’ really means. There are only around 7B people on the planet and so its not even clear what it means without some strange superpopulation.

I [Gary] assume that the number computed is the probability that 2 people other than bin Laden’s parents having sex will produce (have produced) a kid with the same DNA as the person shot in that house in Pakistan. But that’s entirely different than the probability of a false positive. To compute that, you have to include the probability that bin Laden had a twin (presumably small but way way larger than 1 in 11 quadrillion), the probability that the CIA had the wrong DNA on file for bin Laden (that’s got to be higher than 1 in a thousand; the guy’s been a marked man for decades!), the probability that there wasn’t a lab test error (no where near zero and estimable and probably knowable), the probability that no one was lying, etc., etc.

Maybe this is just a blog post, but I think that the whole field of forensics ought to stop continually repeating this nonsense, and that probably takes a paper.

I suspect people must have written about this very issue as it relates to DNA testing, but I don’t know the literature. I do remember that Mosteller and Wallace, in their classic book on classifying the Federalist Papers, discussed the idea of “outrageous events” (I think that’s the term they used) and why it didn’t make sense for them to talk about p-values of 10^{-6} or whatever.