This eagle owl is pictured at a zoo in Mulhouse, eastern France. (Sebastien Bozon/AFP via Getty Images)
Good science is defined, in part, by the fact that it can be replicated by other scientists. To bring more of this rigor to political science, at least 27 journal editors have agreed to implement the Data Access and Research Transparency (DA-RT) guidelines. DA-RT will “require authors to ensure that cited data are available at the time of publication through a trusted digital repository.” Some political scientists, including Thomas Leeper and Nicole Jenz, have defended DA-RT.
Not everyone is on board, however. Last fall, nearly 1,200 political scientists, including 10 former American Political Science Association (APSA) presidents, signed a petition asking the editors to delay implementation of the guidelines. Some DA-RT opponents are concerned that “meaningful transparency is impossible” in some types of qualitative research–research that relies on humans as much as on numbers.
To protect the privacy of their subjects and sources, for example, scholars may not be able to reveal the names or other identifying information about people they interview for their research. This makes reproducing those interviews impossible.
Some argue that the DA-RT standards for qualitative work are unclear and that this could deter or prevent qualitative scholars from publishing their research. Jarrod Hayes goes so far as to argue that qualitative work already exhibits greater analytical transparency than quantitative work.
[Political scientists are debating how to make their research more transparent. Here’s a way forward.]
The real divide in sharing data is not between two groups of scholars but between two kinds of research
To some observers, the DA-RT petition emerges out of and exacerbates a divide in the discipline between qualitative and quantitative scholars. Rather than coming down on one side or another in the DA-RT debate, we sought to better understand how scholars actually engage in data sharing. Who shares data? Who shares what kind of data? Is the split over data sharing actually between qualitative and quantitative scholars?
We use the results of the 2014 Teaching, Research and International Policy (TRIP) survey of 1,615 international relations (IR) scholars to explore views on data sharing in the discipline. This survey includes 38 percent of all IR scholars in the United States (broadly defined to include anyone who teaches or does research on a political issue that crosses national boundaries). The margin of error for the survey is plus or minus 1.91 percent.
We found that the discipline’s current divide is not between those scholars who primarily use statistical methods and those who chiefly employ qualitative approaches. Rather, IR scholars—regardless of whether they consider themselves to be primarily quantitative or qualitative researchers—are less likely to share their qualitative data than they are to share their quantitative results.
In separate questions, the TRIP survey – which was fielded from September to November 2014, when the discussion of DA-RT was active in some circles, but well before the petition drive and the concurrent spate of blog posts about the APSA initiative–asked respondents if, in the past five years, they had shared quantitative and qualitative data that they had produced.
Researchers could indicate whether they had made all of their data available, had made some of their data available, had not yet made available their data but intended to do so after publication, or did not intend ever to share the data. “Sharing” was defined in the survey as “given to other scholars, deposited in an archive, posted online or otherwise made available.”
As the figure below shows, IR scholars who generate quantitative data are more likely than those who generate qualitative data to say that they have shared all or some of their data. (Discussion of other potential explanatory factors, including age and academic rank, can be found here.)
This finding – that IR researchers share their quantitative data more often than their qualitative data – may not seem surprising, since many leading journals already require authors to share their computational data and models for replication purposes.
[Here’s why I’m skeptical about the new “transparency” standards.]
But data sharing could mean more than simply what journals require or expect for quantitative data. To measure these other data-sharing arrangements, the TRIP survey included a follow-up question for those scholars who said that they had shared their data in the past five years, asking them to select all means by which they had granted others access to their data.
Sixty-one percent of respondents who shared qualitative data and 56 percent of respondents who shared quantitative data indicated that their sharing included making “ad hoc arrangements with other scholars.” Even quantitatively oriented scholars were more likely to informally share data with other scholars than to make it publicly available through a journal.
Do our findings reflect a divide in the discipline between qualitative researchers, who don’t share their data, and quantitative scholars, who do? Not necessarily. The TRIP survey asks respondents about the primary method and any secondary methods they employ. Nearly 87 percent of IR scholars report using more than one methodological approach in their research.
If we look at how these multi-method IR scholars report sharing their quantitative data, a gap emerges between researchers who identify as primarily quantitative and those who identify as primarily qualitative.
As the figure below shows, more than half of scholars who identify their primary research method as quantitative report that they shared all the quantitative data they produced over the past five years, and roughly another quarter made portions of their data available.
When asked about their qualitative data from the previous five years, most of both quantitatively and qualitatively oriented researchers report that they will not share access. Although the margins are small, quantitative scholars even reported that they were slightly less likely than qualitative researchers to share all or some of their qualitative data.
What about individually? The graph below plots the overlap between responses to the question about sharing quantitative data and responses to the question about sharing qualitative data. For example, the lower left cell plots the percentage of those respondents who say that they have made all of their quantitative data available and who also say that they have made all their qualitative data available. Each cell in the graph is shaded according to the proportion of the sample (row) that it represents.
The concentration of relatively darker cells in the lower right of the graph shows that many researchers who produce both quantitative and qualitative data are less likely to report that they had shared or intend to share their qualitative data.
Of those respondents who said they had shared all of their quantitative data, only 38 percent indicated that they had made all their qualitative data available, and 27 percent said they did not intend to make their qualitative data available at all.
Individually, therefore, many IR scholars who report that they use several research methods also report that they have given or will give other scholars less access to their qualitative data than to their quantitative data.
Our findings do not speak directly to the DA-RT debate, but they do suggest that opposition to the new requirement that all data be made available online should not be dismissed as simply qualitative scholars’ resistance to increased transparency.
Rather, as the results of the TRIP survey suggest, both qualitative and quantitative researchers currently make very different decisions about whether to share their quantitative and qualitative data.
Right now (rightly or wrongly) the DA-RT standards, with journal editors’ discretion, are to apply uniformly to different types of data. While scholars using and creating large datasets may understand how reproducibility standards apply, they may not be so clear for scholars using other types of data.
But the new DA-RT procedures will require change by scholars of all methodological stripes, particularly in how they report and share their qualitative data.
Elizabeth Martin is project manager for the Teaching, Research, and International Policy Project, and Susan Peterson is Reves Professor of Government and International Relations, both at the College of William & Mary.
Note: An earlier version stated that the American Political Science Association (APSA) adopted the DART guidelines. That is not accurate although in 2012 the APSA Council approved changes to the ethics guide involving transparency.