Methodology

Meta-analysis, game theory, and incentives to do replicable research

Jan 28 '12

One of the key insights of game theory is to solve problems in reverse time order. You first figure out what you would do in the endgame, then decide a middle-game strategy to get you where you want to be at the end, then you choose an opening that will take you on your desired path. All conditional on what the other players do in their turn.

In an article from 1989, “Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts,” Keith O’Rourke and Allan Detsky apply this principle to the process of publication of scientific research:

From the statistical point of view, there really is no escape from performing a de facto meta-analysis. One can either judge the effectiveness of a therapy based solely on the most recent study and ignore all previous studies, a method which is equivalent to giving the most recent study weight 1.Oand all previous studies weight 0, or try to choose the weights on some scientific basis . . . If important differences in study findings exist they must be identified and explained.

That most researchers realize the need for comprehensive, rigorous, and public peer review stating their results in the context of previous trials is evidenced by the literature review section in almost all scientific articles. Meta- analysis is a further development and refinement of this approach offering a more rigorous and coherent treatment of past research work. It be targeted for funding and support. If this is tempting to propose that no experimental seems surprising, perhaps we need to clarify results should be published without inclusion of what is involved in an appropriate meta-analysis an appropriate meta-analysis. In effect, one as opposed to a simple statistical pooling of the might suggest that a literature review section ought to be based on an explicitly described methodology in place of the usual ad hoc approach.

So far, nothing exceptional. But then O’Rourke and Detsky continue:

What is it about meta-analysis that will actually help bring about improvement in individual research efforts? . . . the comprehensive, rigorous, and public peer review that a meta-analysis entails will encourage high quality participation by members of the research community in the resolution of the inadequacies. . . .

With a better understanding of meta-analysis in the context of the full scientific research process, meta-analysis is seen as a key element for improving individual research efforts and their reporting in the literature. This in turn will further enhance the role of meta-analysis in helping clinicians and policy makers answer clinical questions.

The idea (if I’m reading O’Rourke and Detsky correctly) is that, not only is meta-analysis appropriate for summarizing existing dat, also the threat or promise of meta-analysis provides an incentives for researchers to follow better practices in their new projects. If you know (or think there’s a high probability) that your work will be processed through a rigorous meta-analysis, this motivates you to be careful, to supply replication materials (otherwise your study will get a low weight), etc. That’s where the game theory comes in.