I recently attended a talk by Ted Miguel presenting results from an experimental evaluation of a “community driven development” program in Sierra Leone that he conducted with Katherine Casey and Rachel Glennerster. It’s a terrific and very interesting paper and project, but that’s not what I wanted to post about.
After the talk, a colleague of mine, who was impressed by the paper and who knew that I was involved in a similar randomized evaluation of a “community driven reconstruction” project in Liberia, said something along the lines of “Still, I don’t know, isn’t there something about these experimental evaluations that makes you a little uneasy, kind of like we’re treating these people like rats in a maze?”
I said, in so many words, “yes, definitely. But we tell ourselves it’s for a good cause [which I think it is – figuring out what sorts of development aid actually produce good results], and in any event it’s nothing compared to what the development industry itself is doing on this score.” By which I meant that the little experiments and surveys and measurement devices being used in randomized aid project evaluations are like nothing compared to the billions of dollars of attempted social, political, and economic engineering undertaken every year in donor-funded development projects.
For our little experimental evaluations, we have to have our research approved by university “institutional review boards” (IRBs) that assess whether what we are proposing is ethical in terms of treatment of and risks to the subjects. One often hears grousing about IRBs asking clueless questions or demanding unrealistic procedures for obtaining subjects’ consent for surveys and such. But no one I know thinks IRBs are a bad idea in principle – quite the contrary.
Why is there nothing like an IRB for development projects? Is it that aid projects are with the consent of the recipient government, so if the host government is ok with it then that’s all the consent that’s needed? Maybe, but many aid-recipient governments don’t have the capacity to conduct thorough assessments of likely risks versus benefits for the thousands of development projects going on in their countries. That’s partly why they have lots of aid projects to begin with.
Or maybe there’s no issue here because the major donors do, in effect, have the equivalent of IRBs in the form of required environmental impact assessments and other sorts of impact assessments. I don’t know enough about standard operating procedures at major donors like the World Bank, USAID, DFID, etc, to say, really. But it’s not my impression that there are systematic reviews at these places of what are the potential political and social impacts of dropping large amounts of resources into complicated local political and social situations.
Maybe this is a bad question to ask because we aren’t likely to be able to even guess at all the political and social impacts of specific aid projects, and we don’t want a too-rigorous application of “Do No Harm” to stop us from trying to do anything positive at all.
That could be right, but I’m still struck by contrasts like the following: For a survey conducted as part of an aid-program evaluation, the IRB process will ask a researcher if any survey questions might cause discomfort or recollection of highly unpleasant events for the respondent, and if so what are the procedures for preventing this and whether the potential benefits outweight the potential risks. By contrast, a specific aid project could, by introducing a pile of valuable resources, lead to intra- or inter-village competition that might, for all we know, contribute to people getting killed. Those risks might be outweighed by the potential benefits for most projects, but are there any institutionalized procedures for asking the question in the development sector? Should there be?
By the way, leaving aside the question of IRBs for aid projects, the World Bank and other major donors carry out a great deal of research that involves collecting data on, and sometimes manipulating, human subjects. Again I might be wrong, but as far as I know this research doesn’t need or get something like IRB approval. If you think it’s a good idea for academic researchers, what’s the argument for why donor research would be different?