The Problems with Studying Civilian Casualties from Drone Usage in Pakistan: What We Can’t Know

by Erik Voeten on August 17, 2011 · 6 comments

in Blogs

We are delighted to have a guest post from my colleague C. Christine Fair, whose research wehaveblogged about before. Christine is an assistant professor at Georgetown University’s Security Studies Program. She is the author of The Madrassah Challenge: Militancy and Religious Education in Pakistan and co-editor of  Treading on Hallowed Ground: Counterinsurgency Operations in Sacred Spaces.

***

Drone opponents have been given a fillip by two recent heroic efforts. Both the Bureau of Investigative Journalism (BIJ) as well as Peter Bergen and Katherine Tiedemann of the New America Foundation (NAF) have sought to assemble databases of drone strikes and their outcomes. Unfortunately, despite the best of intentions, both initiatives and derivative analyses suffer from similar empirical weaknesses that obfuscate the debate that surrounds the drone program in Pakistan as much—if not more—than they elucidate the same.

The BIJ reports that 160 children are among the 2,292 people reportedly killed by US drones since 2004. In the same article, an un-named U.S. counterterrorism official who was interviewed by the team estimated that 2,050 people have been killed in drone strikes of whom only 50 are non-combatants. Both figures are substantively different from tabulations produced by NAF, which claims that between 2004 and 2010, between “830 and 1,210 individuals, of whom around 550 to 850 were described as militants in reliable press accounts, about two-thirds of the total on average. Thus, the true civilian fatality rate since 2004 according to our analysis is approximately 32 percent.”  Which of these three divergent reports is closer to the truth? One thing is certain: none can be disconfirmed or confirmed with the available evidence despite claims of NAF and BIJ.

Their methodologies and data are fundamental weaknesses, although neither seem aware of this. Both NAF and BIJ claim that they have assembled a database which covers each individual strike in Pakistan in detail.  Unfortunately, both efforts fundamentally rely upon Pakistani press reports of drone attacks. Both claim that they use non-Pakistani media reports as well. For example the BIJ explains in their methodology discussion that the “…the most comprehensive information on casualties lies in the thousands of press reports of drone strikes filed by reputable national and international media since 2004. Most reports are filed within a day or two of an attack. Sometimes relevant reports can be filed weeks – even years – after the initial strike. We identify our sources at all times, and provide a direct link to the material where possible.”

Similarly, NAF  explains that their database “draws only on accounts from reliable media organizations with deep reporting capabilities in Pakistan, including the New York Times, Washington Post, and Wall Street Journal, accounts by major news services and networks—the Associated Press, Reuters, Agence France-Presse, CNN, and the BBC—and reports in the leading English-language newspapers in Pakistan—the Daily Times, Dawn, the Express Tribune, and the News—as well as those from Geo TV, the largest independent Pakistani television network.” Indeed, BIJ relies upon the NAF sources as the organization states in its methodology explanation.

While these methodologies at first blush appear robust, they don’t account for a simple fact that non-Pakistani reports are all drawing from the same sources: Pakistani media accunts. How can they not when journalists, especially foreign journalists, cannot enter Pakistan’s tribal areas?  Unfortunately, Pakistani media reports are not likely to be accurate in any measure and subject to manipulation and outright planting of accounts by the ISI (Pakistan’s intelligence agency) and the Pakistani Taliban and affiliated militant outfits.

Pakistani journalists have readily conceded to this author that perhaps as many as one in three journalists are on the payroll of the ISI. In fact, the ISI has a Media Management Wing which manages domestic media and monitors foreign media coverage of Pakistan.  Even a prominent establishment journalist,Ejaz Haider, has questioned “What right does this wing have to invite journalists for ‘tea’ or ask anyone to file a story or file a retraction? The inquiry commission [to investigate the death of slain journalist Shehzad Saleem] should also look into the mandate of this wing and put it out to pasture.”

Pakistani journalists have explained to this author that, with respect to drone strikes, either the Pakistani Taliban call in the “victim count” or the ISI plants the stories with compliant media in print and television—or some combination of both.  In turn, the western media outlets pick up these varied accounts. Of course the victim counts vary to give the illusion of authenticity, but they generally include exaggerated counts of innocents, including women and children.  Of course as recent suicide bombings by females suggest, women should not be assumed innocent by virtue of their gender.

Thus, these reports mobilized by NAF and BIJ, despite the claims of both teams of investigators, cannot be independently verified. At best, their efforts reflect circular reporting of Pakistani counts of dubious veracity.

Equally problematic for the BIJ effort is that it aims to focus upon those drone attacks that the organization alleges to have killed innocent civilians. This is bad social science and suffers from “dependent variable bias.” In common parlance, this means that the BIJ team is focusing upon the outcomes that interest its researchers the most—alleged innocent casualties—while ignoring those events that are less interesting—those drone attacks that kill their targets exclusively or mostly. In doing so, the BIJ study focuses upon one alleged aspect of the program while ignoring the overall outcomes of the drone campaign. It gives an extremely lopsided view of the program even if one can make the leap of faith that their data have any validity whatsoever.

Why These Claims Cannot be independently Verified

In Pakistan, there is no civilian casualty reporting unit such as that run by the United Nations Assistance Mission to Afghanistan (UNAMA). UNAMA’s civilian casualty reports draw from multiple sources including on the ground interviews at the site of attack. It also includes casualties perpetrated by the Taliban and other anti-government elements. This is important: increasingly the Taliban have accounted for a larger share of the deaths while overall civilian casualties have increased. Drone accounts rarely consider the victims of the drones’ targets. This is in some measure due to lack of clarity about who the targets were and what their crimes were.

One of the key differences between the theatres of Pakistan and Afghanistan is that in Afghanistan, NATO forthrightly discusses the casualties because the assaults producing them are overt. One can even find video footage of some strikes on the internet. While NATO usually announces their figures after an investigation, the Taliban are usually able to put out an estimate much earlier. These figures often stick even though they tend to grossly exaggerate NATO civilian casualties while underestimating their own. Why would the Pakistani Taliban not do the same having seen the efficacy of this tactic from the Afghan Taliban across the border?

Alas, even if there were such a body authorized by the government of Pakistan to investigate drone operations and their outcomes, it would be nearly impossible to verify claims advanced by opponents and proponents alike. The reasons for this are straightforward if rarely acknowledged by the former.

One problem is that persons who would be interviewed in the Federally Administered Tribal Areas (FATA), where all drone strikes occur, have an enormous incentive to report that the drones’ victims are “innocent,” especially children and women. There are two principle reasons for response bias. First, who would admit that terrorists are living in their midst much less resident in their own home or that of a relative? Such an admission could elicit further drone attacks or even Pakistani operations. Second, FATA is an extremely coercive environment. Even if someone could directly interview an observer of the strikes, why on earth would the interlocutor trust the person asking the question? How could he (or less often she) be certain that their responses would not be reported to the Pakistan Taliban or Pakistan’s intelligence agencies or even American, Israeli or Indian intelligence agencies?  Worse respondents may even believe that the person asking such questions about drone strikes and their outcomes may be directly associated with one or more intelligence agency. While American audiences may find such suspicions about U.S., Israeli or Indian intelligence operating in the region to be absurd, many Pakistanis accept this presence in FATA as a matter of fact.

Opponents’ of drones have criticized this argument as cruel and insensitive, complaining this puts the onus of proof upon the ostensible victims.  However, isn’t this case generally with alleged victimization? In what legal system are claims of victimization taken as fact without evidence against the alleged perpetrator or even evidence that a crime took place?

Opponents’ of drones also point to recent photographic evidence that depicts harm to property and persons purported to be the results of U.S. drone strikes. Take for example photographic exhibition of Noor Behram, a journalist from the North Waziristan (an agency of FATA).  Again, while my counter-argument may unsettle persons who accept these photographs as fact, what proof has Mr. Behram or others presented that the damages to property and injury to persons are due to drone attacks? My intention is not to impugn these likely well-intentioned persons, merely to call attention to the lack of evidence for the claims that drones caused these outcomes. Shrapnel from hundreds of terror attacks as well as airstrikes by Pakistan’s air operations are also possible causes of these varied damages. Indeed, Pakistani air strikes and terror attacks have been far more injurious in terms of deaths, damage and displacement. What evidence do photographers present that these images are not the result of terrorism or Pakistani air strikes?  Shrapnel patterns from drones, terrorism and air strikes are different. Experts should be able to discern probable causes. Who has bothered to examine these images for such purposes?

There are also compelling institutional incentives to keep the drone program cloaked in secrecy that are held by both the Pakistan military and the U.S. CIA.

The Pakistan Military: Drones Expose Its Weakness

The Pakistan military and intelligence agencies are admittedly in a tough spot.  The Pakistan army has long argued that it alone can secure the ideological and physical frontiers of Pakistan, as eloquently described by Husain Haqqani, the current Pakistani Ambassador to the United States. The army has long held that civilians are incompetent stewards of the state and have used the army’s self-proclaimed relative competence to justify the army’s inordinate demands upon state resources and its arrogated right to run the state directly and indirectly since the mid-1950s.

Yet in recent years, the army’s image has been tarnished. Despite the loss of thousands of military personnel, the army has not been able to eradicate the Pakistan Taliban which has ravaged the state and its citizenry.  Pakistan has repeatedly demonstrated that it cannot exert territorial sovereignty, especially over the problematic frontier along the Afghan border.  The U.S. unilateral raid to kill Osama Bin Laden in an army cantonment town near the Pakistan’s premier military academy at Kakul has done little to reassure Pakistanis that their army can do the needful. As one person said to me in Islamabad the week that Bin Laden was killed, “We are either a rogue state or a failed state.” These apprehensions have been fueled by ongoing accounts that the army, navy and air force have sympathizers of terrorists in their ranks who have facilitated terror attacks on Pakistan’s armed forces.

The army has no incentive to admit that it needs the drone program to help it do what it cannot do on its own: kill Pakistani terrorists as well as foreign terrorists on Pakistani soil. Foreign terrorists are also Pakistan’s problem as al Qaeda has long aimed its sites on Pakistan. Indeed, the status quo policy of misreporting the outcome of the drone strikes suits the army well. It allows it to free ride off the program while antagonizing anti-Americanism in Pakistan.  Doing so allows the Pakistan army to refuse to engage in operations that are domestically resisted. In an unvirtuous circle, this resistance is also nurtured by Pakistan’s intelligence agency by asserting that groups like Lashkar-e-Taiba/Jamaat-ud-Dawa is innocent of terrorism charges and by denying the presence of high-value targets on Pakistani soil. Recall that Pakistan claimed that Bin Laden was not in Pakistan for nearly a decade. It continues to deny that Mullah Omar, the leader of the Afghan Taliban, is in Pakistan.

The Central Intelligence Agency Wants to Cover Its Assets Too

I believe that greater transparency about the drones will likely be exculpatory and actually extend the longevity of the drone program.  From my own research, drone attacks are more complex than ordinary air strikes. Drone strikes involve lawyers, intelligence officials, actual pilots and others to assess the nature of the target, establish a pattern of life to avoid civilian casualties, and ultimately to authorize or even call off a strike. Like conventional strikes, they are conducted by actual air force pilots. Unlike conventional air strikes, analysts become familiar with their would-be victims and have to watch the video footage of the strike and assess its outcome. Analysis of such footage also leads to information about other potential targets as affiliated militants often rush to the scene. (Indeed, the United States likely learned this from terrorists who pioneered the tactic of attacking at one site and waiting for first responders to appear only to strike again to maximize casualties.) It is a little known fact that people involved in this program are also vulnerable to post-traumatic stress disorder.

If greater transparency may be a good thing, why doesn’t the CIA acquiesce? After all the U.S. President has been clear that drone strikes will continue. Why deny what is patently obvious?

First is the legal issue that the program is covert per the legal code under which they are conducted. At least one high-level U.S. department of state official has expressed privately his or her interest that this program be overt such that U.S. officials can discuss the outcomes openly with an outraged Pakistani public. Second, the CIA likely does not want to show its cards by revealing the details with which they can visualize events.  Revealing CIA’s capabilities may well give the enemy the insights needed to develop counter measures. Similarly state actors may do the same. Third, these drone strikes involve an elaborate network of Pakistani informants including persons who “chip” targets to direct the drone strike.  The notorious Pakistani terrorist, Baitullah Mehsud, was “chipped” by his own father-in-law. The resultant drone strike killed his own daughter along with her husband, Baitullah Mehsud, while the former was giving him a massage. It would take a new definition of “innocent” to conclude that Mehsud’s wife, an obvious aider and abettor of mass murder, was innocent. These assets cannot be compromised under any circumstances.

However, this author believes that some version of events can be and should be disclosed with adequate measures to attend to CIA’s legitimate concerns.

The Way Forward?

As I have written elsewhere, U.S. officials interviewed as well as Pakistani military and civilian officials have confirmed to this author that drones kill very few “innocent civilians.” Indeed, it was these interviews that led me to revise my opinion about the drone program: I had been a drone opponent until 2008. I know believe that they are best option.

In March of 2011,  Maj. Gen. Mehmood Ghayur of the army’s Seventh division in North Waziristan explained to several journalists in Peshawar that the drone program was very effective noting that “myths and rumors about U.S. Predator strikes and the casualty figures are many, but it’s a reality that many of those being killed in these strikes are hardcore elements, a sizable number of them foreigners.”  This caused some rankling among the army’s top brass. Unsurprisingly, the spokesman for Pakistan’s armed forces, Maj. Gen. Athar Abbas, clarified that Ghayur’s remarks reflected his “personal assessment” and further opined that they were reported without proper context.

What is clear is that the strategic effects of this program create more enemies. Opponents and proponents of drones alike generally agree on this point. Drone proponents however would counter that Pakistanis already hate Americans and thus the marginal hatred caused by them is negligible. After all, you cannot get more Anti-Americanism in Pakistan than there is at present. Drone proponents also credit the program with putting intense pressure on al Qaeda and diminishing its ability to operate. Thus the drones offer strategic advantages that mitigate in. some measure. the costs.

Nonetheless, die hard opponents focus on the strategic costs, deny strategic benefits and assert that the drone program offers tactical benefits at best. They argue that the solution to what they see as a strategic problem is a cessation of the program in Pakistan if not elsewhere.

I have a different approach. First, Pakistan should cease the propaganda campaign against the program. U.S. officials have made this request, thus far to no avail. Second, the United States should counter the campaign with facts about Pakistani complicity and the details of the attack and its victims. This requires transparency about who the victims were, what they did, how the U.S. determined their guilt, with what degree of Pakistani complicity did the attack occur, who were their Pakistani victims among other aspects of the drone attack.

Needless to say a fundamental question here is what defines “innocent”? What defines a “justifiable target.” Even if we believe some version of the U.S. official narrative that they kill few if any “innocents,” would there be domestic agreement in the United States and Pakistan about what “innocent” and “guilty” means? What evidence threshold exists for such claims? Do these definitions comport with international norms?

Drone attacks (and oddly even suicide bombing) are subject to the same rules of war as other weapon systems under international humanitarian law (IHL). IHL does not proscribe a weapon if the intent is not to kill preponderantly innocent civilians or if the outcomes of its use do not disproportionately result in civilian deaths. Nevertheless the above questions are all legitimate. But they cannot be answered with both the government of Pakistan and United States colluding in this conspiracy of misrepresentation and denial.

If the drone program is going to continue in Pakistan and elsewhere, domestic and international concerns will need to be allayed. There is only one way to do that: expose the program to light of day and analytical scrutiny.

 

{ 6 comments }

Scott Monje August 17, 2011 at 10:40 am

I’d like to thank you for a very thought-provoking post. I’m a bit skeptical about the proposal to have the CIA give that much detail about its targeting. The revelation of sources and methods (along with domestic police operations) is among the few things expressly prohibited in its charter, not to mention its institutional culture. I also still wonder about their capacity to identify terrorists through aerial photography, although I admit to knowing nothing about the process. I would like to ask for one point of clarification. In the following excerpt, are you suggesting that they attack the same site twice on the assumption that anyone who rushed to the scene is an affiliated militant?

“Analysis of such footage also leads to information about other potential targets as affiliated militants often rush to the scene. (Indeed, the United States likely learned this from terrorists who pioneered the tactic of attacking at one site and waiting for first responders to appear only to strike again to maximize casualties.) “

Raimo Kangasniemi August 17, 2011 at 3:17 pm

I think this article is very, very weak in it’s arguments that the “innocent civilian casualties” – like there would be “guilty civilian casualties”? – are low. Just claims that unnamed people have told the writer so and that journalists who claim otherwise must be on ISI payroll, that finding out the truth is hard etc. No evidence, more like desperate wishful thinking if one is generous. One can argue about the number of civilian casualties, agree that finding out the exact number might be impossible and finding out even the correct range of casualties hard, but this kind of argumentation is either extremely naive or dishonest and clumsy.

idiot August 17, 2011 at 4:13 pm

And I’d suggest that any allegations of ISI-taint should likely be countered by fact that civilian causalities from these drone attacks are already low already according to these so-called “Pakistani sources”. NAF reports that “only” 32% dead happen to be classified as civilians while an anonymous counter-intelligence offer quoted by the BIJ reported only a 2% civilian death toll (50/2050). If an individual wanted to discredit the drone program, then one has to offer a civilian death toll higher than that 2%-32%.

In fact, the debate about the ISI taint seems to imply to me that there’s an UNDER-reporting bias of how many civilians are being killed (and it’s being done by the ISI to try and makes the drones seem less horrible to the Pakistani population), but the truth is casualty estimates are going to be just that, estimates.

Chris Woods August 18, 2011 at 9:07 am

I lead the Bureau of Investigative Journalism’s team on the US covert war here in London. You state that ‘[Our] methodologies and data are fundamental weaknesses, although neither seem aware of this…’

Actually we are aware of the limitations of using press reports alone (though nevertheless they are a critically important source). We have attempted to combat that. As our Methodology makes clear, we also incorporate a host of other materials in our work. Leaked intelligence documents and US government cables; case studies provided by lawyers; field reports by NGOs; and the writings of journalists, academics, politicians and former intelligence officers, for example, all actively inform our work. We have deployed field researchers in Waziristan on one occasion so far. We have even incorporated the CIA’s own critique of our writings on four of its strikes (and have urged them to offer a similar response to the other 288). We do think our study offers the clearest public understanding of what is taking place.

I agree that there are multiple biases at play in Waziristan. Nevertheless, when we used our own field researchers in Waziristan, we found a reasonable correlation between what was originally reported and what we were able to discover on the ground. And independent reporting matters. I would suggest that had the Bureau and the New York Times not published their work last week, the Agency would not have released its estimates of 2050 killed in the strikes. An absence of public scrutiny – particularly when elements of a war are being conducted by secret and unaccountable intelligence agencies – is dangerous.

We also ‘do not aim to focus on civilian casualties’. We do aim to identify, as part of our broader work, those incidents in which civilian deaths are credibly reported. There is a difference. Possible evidence of civilian deaths is of great value to policy makers (or why else would the CIA contest such figures so aggresssively?).

You might want to take a look at our recent work on a specific US claim that it had not killed a civilian in the past year. At that time we understood this to mean no civilian deaths since August 23 2010. The Agency has since pushed the date back to May 2010.

We identified 25 cases in total of concern after extensive research. Of these, ten cases clearly indicated around 45 civilian deaths in the time period. We were able to supply most of the names of those killed, including six children, following the work of our field researchers in Waziristan. ‘Civilian’ is clearly defined in international law. I suspect that the CIA is significantly testing that definition right now, given its continuous insistence of ‘zero casualties’ in the face of overwhelming evidence to the contrary.

We certainly do not ‘ignore less interesting attacks’ – indeed if you examine each of our strike reports, you will see we present all available evidence, and challenge previous media assumptions. We reduce civilian casualty estimates where our researchers find otherwise. We make clear where any civilian death report is contested, hence the wide range for our reported civilian death toll (385-775). And we challenge assumptions. For example, when a doctor was killed in Waziristan in March 2008, media at the time reported his death sympathetically. We link to the al Qaeda video later put out in his honour, showing the good doctor wielding a gun at an AQ training camp.

Your own view that drones kill ‘very few innocent (?) civilians’ is a contested one. I recently met with a senior Pakistan military official who told me he believes ‘civilians are killed on most occasions, but that isn’t the point’. It is in the interest of both US and Pakistani officials to play down civilian casualties, it is safe to assume. The speech made by Gen Ghayur is certainly a valuable perspective, yet occurred just days before the strike on a jirga killed between 19 and 40 civilians on March 17 this year. That, I’m told, is why his speech is now not referred to in polite company.

I agree with you that drones should be open to more scrutiny and honesty from those involved, and fully subject to IHL. But you imply that the only credible source for such information can be the agencies conducting the attacks. As a journalist who has covered conflicts directly for many years, I have found that honesty from those persecuting wars is a rare and precious commodity. The Bureau’s work, along with that of NAF and others, remains a necessity.

Scott Monje August 18, 2011 at 10:07 am

The problem of identifying civilians and avoiding injury to them is going to remain a perplexing one, even if the term civilian is defined in international law. One could easily argue that they are all civilians. Raimo Kangasniemi appropriately asks whether there are guilty civilians. Yes, Baitullah Meshoud was one. He was an armed civilian participating in an insurgency. Merely being an armed civilian would not have distinguished him from many other people in the region. The distinction depends on actions and intentions, criteria, it seems, that would be difficult to identify from 30,000 feet up. Whether that means his wife was actively aiding and abetting the insurgency is not so clear. Are all spouses guilty? All spouses who stay with their families? Who give massages? On the other hand, can insurgents avoid attack forever by remaining among noncombatant civilians, whether that attack be from the air or from the ground? As I recall, when U.S. troops used to patrol the streets of Mogadishu, militants would toss handgrenades from behind crowds of bystanders. How to respond is a real dilemma, and the answer will never be straightforward.

Phil Sheehan August 21, 2011 at 3:44 pm

This whole set of navel-gazing quotes and statistics and analyses derives from a premise I cannot accept: that regardless of quibbles over reporting accuracy, kill efficiency, and costs, we have good and sufficient reason to drop bombs on rural Pakistan.

Comments on this entry are closed.

Previous post:

Next post: