Archive | Methodology

NSF-sponsored research on deep interactions

As part of our series on recent NSF-funded political science research, Rick Wilson points to . . . Yair Ghitza and Andrew Gelman. (Forthcoming). “Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Groups.” AJPS [DOI: 10.1111/ajps.12004].

Rick writes:

Accurately measuring attributes of the American public is critical to the success of government and society. This includes polls that allow lawmakers to understand their constituents’ preferences to inform their policy choices. It is also indispensible for government to accurately measure unemployment or the population itself through the Census. Yet conducting and analyzing surveys is an increasingly difficult proposition, with the growing abundance of cell-phone-only populations, the difficulty in reaching certain segments of the population, and many other challenges. New measurement techniques need to be developed for the 21st century.

This research demonstrates that standard survey analysis methods are often unstable and unsuitable for estimating values for small populations. In turn the article introduces a statistical method that combines survey and Census information that overcomes these problems. This particular research focuses on vote preferences, but the method can be applied generally to survey analysis, and as such is valuable for any person or organization with an interest in accurately measuring aspects of the American public.

This work should be viewed as a statistical innovation driven by an interdisciplinary approach to research. In a world where “big data” are becoming increasingly available, there is a temptation to think that standard statistical and computational methods can be arbitrarily pointed at large piles of data to make sense of the world; this thinking implies that funding should only go to the hard STEM disciplines, which are traditionally better trained at handling these quantitative tasks. This research demonstrates that standard data-crunching approaches often miss important structures, or can be entirely inappropriate for the question being asked. The research makes clear that scientific progress is achieved through collaboration across disciplines, such as between technical experts such as statisticians and computer scientists, and domain-specific experts such as political scientists, education policy experts, medical researchers and the like.

Continue Reading

The relevance of statisticians to researchers in different fields of social science

Someone pointed me to a remark someone who felt that statisticians were not doing their job to help out “mathematically challenged” psychology researchers. My first thought was that statisticians often help in an indirect way, by developing advanced methods that quantitative psychologists then translate to their colleagues, but I also realized that there was some specific advice I could give that could be used right away. This made me think that my colleagues and I should put together a short document (an article? webpage? wiki? pamphlet?) of statistical advice. Maybe 50 useful tips. Much of this is in our books but it could be useful to have something that people can use right away, with no prerequisites and without feeling that it would be a big time commitment.

The thing I wanted to talk about here on the Monkey Cage, though, is that I can’t imagine a political scientist complaining that statisticians weren’t helping them out. The difference, I suppose, is that psychology has a longstanding field of psychometrics and mathematical psychology, and these people have been developing their own methods for many decades. In contrast, political methodologists take from other fields (mostly econometrics and statistics), so they are used to having to learn the language of others. And political scientists who are not methodologists (the equivalents of the “mathematically challenged psychologist” who asked the original question) know that if they want to do statistics, they have to learn some statistics; they don’t really expect to get by otherwise.

P.S. Just to be clear: None of this is meant to disparage qualitative research. Qualitative research is great. I’m certainly not saying that all researchers need to use statistics. This discussion is all about people who are doing quantitative work and fell the need to use methods that are somewhat beyond their mathematical/statistical comfort zone.

Continue Reading

Elections expert Jim Snyder sez: RD is A-OK

Andrew Eggers, Olle Folke, Anthony Fowler, Jens Hainmueller, Andrew Hall, and Jim Snyder write:

Many papers use regression discontinuity (RD) designs that exploit “close” election outcomes in order to identify the effects of election results on various political and economic outcomes of interest. Several recent papers critique the use of RD designs based on close elections because of the potential for imbalance near the threshold that distinguishes winners from losers. In particular, for U.S. House elections during the post-war period, lagged variables such as incumbency status and previous vote share are significantly correlated with victory even in very close elections. This type of sorting naturally raises doubts about the key RD assumption that the assignment of treatment around the threshold is quasi-random. In this paper, we examine whether similar sorting occurs in other electoral settings, including the U.S. House in other time periods, statewide, state legislative, and mayoral races in the U.S., and national and/or local elections in a variety of other countries, including the U.K., Canada, Germany, France, Australia, India, and Brazil. No other case exhibits sorting. Evidently, the U.S. House during the post-war period is an anomaly.

In case you got lost somewhere during that paragraph, here it is again:

Across more than 40,000 closely contested elections in many different electoral settings, we find no systematic evidence of sorting around the electoral threshold. Conditional on being in a very close election, incumbents are no better at winning than challengers. . . . the sorting and imbalance that has been discovered in the U.S. House is most likely a statistical fluke . . . Recent concerns about the validity of electoral RD designs appear overblown, as we find no evidence that the underlying assumptions are categorically unsound.

Got it? It makes sense to me.

Continue Reading

Updated survey of small business attitudes

Sander Daniels sends along this info on a survey of small business attitudes in the U.S. A discussion of their methods is here. As I wrote when linking to their survey last year, I don’t know what to make of all of this—who knows what to make of their sample or the responses to these questions?—but I’m impressed that they seem to describe exactly what they did.

Continue Reading

A graph at war with its caption. Also, how to visualize the same numbers without giving the display a misleading causal feel?

Kaiser Fung discusses the following graph that is captioned, “A study of 54 nations—ranked below—found that those with more progressive tax rates had happier citizens, on average.”


As Kaiser writes, “from a purely graphical perspective, the chart is well executed . . . they have 54 points, and the chart still doesn’t look too crammed . . .” But he also points out that the graph’s implicit claims (that tax rates can explain happiness or cause more happiness) are not supported.

Kaiser and I are not being picky-picky-picky here. Taken literally, the graph title says nothing about causation, but I think the phrasing implies it. Also, from a purely descriptive perspective, the graph is somewhat at war with its caption. The caption announces a relationship, but in the graph, the x and y variables have only a very weak correlation. The caption says that happiness and progressive tax rates go together, but the graph uses the U.S. as a baseline, and when you move from the U.S. point on the graph to the right-hand side (more progressive taxes), you see a lot more points below the line than above the line. Thus the visual impression of the graph is that more progressive taxes will lead to lower happiness—-the opposite of the message from the caption.

What can be done here?

I don’t exactly think the graph is “bad data,” and, although the graph says little directly about causation, the data have some relevance to our understanding of policy debates over taxes. If nothing else, we learn that tax progressivity and average happiness some variation among countries. I think a start would be to reframe and put happiness on the x-axis and the tax system on the y-axis, which would allow us to see that, at any happiness level, there is a range of tax systems. with none of the very happiest countries having flat taxes.

Better still might be to make a line plot with three columns: First, a list of country names, in decreasing order from richest to poorest (using, for example, per-capita GDP (yes, I know, such data aren’t perfect!)), then a column showing tax progressivity (if that’s the measure they want to use), then a column showing average happiness.

The advantage of this pair of dotplots is that you get to see the spread in each of these variables with respect to a natural measure (how rich the country is), and there’s no implicit causal story getting in the way.

Continue Reading

Giving credit where due

Gregg Easterbrook may not always be on the ball, but I 100% endorse the last section of his recent column (scroll down to “Absurd Specificity Watch”).

Earlier in the column, Easterbrook has a plug for Tim Tebow. I’d forgotten about Tim Tebow.

Continue Reading

“What Politicians Believe About Their Constituents: Asymmetric Misperceptions and Prospects for Constituency Control”

Jeremy Fox points us to a paper by David Broockman and Christopher Skovron, who look at legislators’s perceptions of their constituents and compare to estimates of the the actual issue attitudes of people living in their districts. Broockman and Christopher Skovron find,

There is a striking conservative bias in politicians’ perceptions, particularly among conservatives: conservative politicians systematically believe their constituents are more conservative than they actually are by over 20 percentage points, while liberal politicians also typically overestimate their constituents’ conservatism by several percentage points. A follow-up survey demonstrates that politicians appear to learn nothing from democratic campaigns or elections that leads them to correct these shortcomings.

They conclude:

These findings suggest a substantial conserva- tive bias in American political representation and bleak prospects for constituency control of politicians when voters’ collective preferences are less than unambiguous.

I have not read the paper in detail, but I was happy to see that they have cool graphs and they use Mister P. So I like that.

Continue Reading