Gary King writes:

This article in the Washington Post quotes an intelligence official that “The chance of a false positive from the DNA testing [that it was bin Laden] is “approximately one in 11.8 quadrillion.” I think the people in this field haven’t the slightest idea what 11 quadrillion really means or what the claim of a ‘false positive’ really means. There are only around 7B people on the planet and so its not even clear what it means without some strange superpopulation.I [Gary] assume that the number computed is the probability that 2 people other than bin Laden’s parents having sex will produce (have produced) a kid with the same DNA as the person shot in that house in Pakistan. But that’s entirely different than the probability of a false positive. To compute that, you have to include the probability that bin Laden had a twin (presumably small but way way larger than 1 in 11 quadrillion), the probability that the CIA had the wrong DNA on file for bin Laden (that’s got to be higher than 1 in a thousand; the guy’s been a marked man for decades!), the probability that there wasn’t a lab test error (no where near zero and estimable and probably knowable), the probability that no one was lying, etc., etc.

Maybe this is just a blog post, but I think that the whole field of forensics ought to stop continually repeating this nonsense, and that probably takes a paper.

I suspect people must have written about this very issue as it relates to DNA testing, but I don’t know the literature. I do remember that Mosteller and Wallace, in their classic book on classifying the Federalist Papers, discussed the idea of “outrageous events” (I think that’s the term they used) and why it didn’t make sense for them to talk about p-values of 10^{-6} or whatever.

According to the NYT: http://www.nytimes.com/2011/05/03/science/03dna.html?_r=1&partner=rss&emc=rss

Their best guess is that they compared to samples from multiple half-siblings (so, potential 1/4 match with each one). But even with a large number of those, “1 in 11 quadrillion” is still a pretty tall claim.

Don’t forget P(X=government telling the truth), which if use trust in government as a prior turns it into a coin toss at best.

At one in 11.8 quadrillion, as my Princeton colleague Ed Turner likes to say, “it is more likely that we are living in the Matrix”.

Remember John Lanchester’s book ‘Whoops’ where he revealed that – before the credit crunch – the banks used fancy mathematical models which suggested that a substantial decline in US house prices would not happen within the lifetime of the galaxy …