Author Archive | Andrew Gelman

Is coffee a killer? The statistical significance filter strikes again

coffee Thomas Lumley writes:

The Herald has a story about hazards of coffee. The picture caption says
Men who drink more than four cups a day are 56 per cent more likely to die.

which is obviously not true: deaths, as we’ve observed before, are fixed at one per customer.  The story says
It’s not that people are dying at a rapid rate. But men who drink more than four cups a day are 56 per cent more likely to die and women have double the chance compared with moderate drinkers, according to the The University of Queensland and the University of South Carolina study.

What the study actually reported was rates of death: over an average of 17 years, men who drink more than four cups a day died at about a 21% higher rate, with little evidence of any difference in men.  After they considered only men and women under 55 (which they don’t say was something they had planned to do), and attempted to control for a whole bunch of other factors, the rate increase went to 56% for men, but with a huge amount of uncertainty. Here are their graphs showing the estimate and uncertainty for people under 55 (top panel) and over 55 (bottom panel) FPO-1 There’s no suggestion of an increase in people over 55, and a lot of uncertainty in people under 55 about how death rates differed by coffee consumption. In this sort of situation you should ask what else is already known.  This can’t have been the first study to look at death rates for different levels of coffee consumption. Looking at the PubMed research database, one of the first hits is a recent meta-analysis that puts together all the results they could find on this topic.  They report
This meta-analysis provides quantitative evidence that coffee intake is inversely related to all cause and, probably, CVD mortality.

That is, averaging across all 23 studies, death rates were lower in people who drank more coffee, both men and women. It’s just possible that there’s an adverse effect only at very high doses, but the new study isn’t very convincing, because even at lower doses it doesn’t show the decrease in risk that the accumulated data show. So. The new coffee study has lots of uncertainty. We don’t know how many other ways they tried to chop up the data before they split it at age 55 — because they don’t say. Neither their article nor the press release gave any real information about past research, which turns out to disagree fairly strongly.

I agree.  Beyond all this is the ubiquitous “Type M error” problem, also known as the statistical significance filter:  By choosing to look at statistically significant results (i.e., those that are at least 2 standard errors from zero) we’re automatically biasing upward the estimated magnitudes of any comparisons.  So, yeah, I don’t believe that number. I’d also like to pick on this quote from the linked news article:
“It could be the coffee, but it could just as easily be things that heavy coffee drinkers do,” says The University of Queensland’s Dr Carl Lavie. “We have no way of knowing the cause and effect.”

But it’s not just that.  In addition, we have no good reason to believe this correlation exists in the general population. Also this:
Senior investigator Steven Blair of the University of South Carolina says it is significant the results do not show an association between coffee consumption and people older than 55. It is also important that death from cardiovascular disease is not a factor, he says.

Drawing such conclusions based on a comparison not being statistically significant, that’s a no-no too.  On the plus side, it says “the statistics have been adjusted to remove the impact of smoking.”  I hope they did a good job with that adjustment.  Smoking is the elephant in the room.  If you don’t adjust carefully for smoking and its interactions, you can pollute all the other estimates in your study. Let me conclude by saying that I’m not trying to pick on this particular study.  These are general problems.  It’s just helpful to consider them in the context of specific examples.  There are really two things going on here.  First, due to issues of selection, confounding, etc., the observed pattern might not be real.  Second, even if it is real, the two-step process of first checking for statistical significance, then taking the unadjusted point estimate at face value, has big problems because it leads to consistent overestimation of effect sizes.

I’m posting this here (as well as on our statistics blog) because I think these points are relevant for political science research as well.

Continue Reading

Larry Summers and Starbucks: can we understand this category more generally?

Reading Sarah Binder’s post on the withdrawal of Lawrence Summers from consideration for a post on the Federal Reserve, I was reminded of our discussion from a few years ago about the similarities of Summers and Starbucks, both of which that are disliked by the left for being too corporate and disliked by the right for being too left-wing.

This made me wonder about the more general category, what other people and institutions have that public-opinion profile. I can’t think of a lot of examples. Bill Clinton, for instance, could have ended up this way, but my impression is that he is liked (even if not loved) on the left, that liberals excuse his centrism because he got a lot done. Similarly, sure, some conservatives were annoyed at the two George Bush for various reasons, but overall I think there’s a clear partisan divide. Nothing like Summers, who was actively opposed by liberal Democrats and conservative Republicans.

Continue Reading

The writing style of aggressive business gurus

Co-blogger Henry Farrell just published an article about what he calls “our new breed of cyber-critics,” arguing that to understand their typically tech-optimistic content we need to understand their economic incentives. In the current era, Henry argues, the economic point of writing is not to get a well-paid job as a journalist or to sell lots of books but rather to make money giving paid lectures. Thus there are various large niches for writers/performers who celebrate big business and rich people (after all, these are the people who pay for most of these lectures) and various small niches for provocateurs (“trolls”) who build a name for themselves by eliciting angry reactions from public figures. Farrell is echoing an argument made by Stephen Marche in the context of celebrity historian Niall Ferguson.

This all makes sense to me, even though the argument doesn’t quite apply in my own case: I’m among the subset of intellectuals who write books and articles on the side, as an adjunct to a steady job. I do give talks at corporate events but, maybe because I don’t rely on that extra money, I haven’t felt the need to adapt my message to what I think the audience wants to hear. I’m not saying that I’m more virtuous than these other people; I’m just in a different circumstance. And, on the substance, I find Henry’s argument persuasive: earlier I objected to one of Seth Godin’s stories as being “bit too glib in treating pop success as some sort of moral arbiter, a kind of Santa Claus that punishes the bad and rewards the good.”

But here I want to talk about something slightly different, which is the annoying (to me) style of these internet business gurus: they’re always getting in your face, telling you how everything you thought was true was wrong. I can’t take being shouted at, and I get a little tired of hearing over and over again that various people, industries, etc., are dinosaurs (even if, sometimes, they are). My guess is that this aggressive style is coming from the vast supply of “business books” out there. These are books that are supposed to grab you by the lapel and never let go.

I also wonder whether there is a stylistic difference: Journalists tell stories; bloggers rant, hector, and explore. Professional journalism is closed; internet writing is open. A newspaper or magazine article is supposed to come to a pat conclusion and to demonstrate certainty. An online article can demonstrate certainty—-and, when it does, you often get that obnoxious over-certainty of Jarvis/Greenspun/etc—-but it can also reflect uncertainty and a search for truth, something you don’t find much in the professional press. Those annoying internet gurus are in some way combining the style of blogging with the assumed certainty of journalism, the worst of both worlds.

Anyway, I wonder what Henry’s thoughts are on all this, perhaps he has some insight on the connections between the style and the content of internet business writing.

Continue Reading

Don’t be so quick to place politicians’ views of “national interests” above the mood of the public

In a news/opinion article entitled, “What Obama Really Thinks About Syria: The president’s TV interviews reveal the naked truths behind his posturing,” William Saletan writes:

Public opinion trumps national interests. When asked whether he would strike Syria without congressional support, Obama told ABC: “Strikes may be less effective if I don’t have congressional support and if the American people don’t recognize why we’re doing this. So I haven’t made a final determination in terms of what next steps would be.” On NBC, he said he would lobby Congress, deliver a TV address Tuesday night, and “I’ll evaluate after that whether or not we feel strongly enough about this that we’re willing to move forward. … I’ve made my decision about what I think is best for America’s national interests, but this is one where I think it’s important for me to pay close attention to what Congress and the American people say.” That sounds like a bald admission that he’s willing to let public opinion override national interests.

But we should be careful about so quickly opposing “public opinion” and “national interests.” As political scientist Benjamin Page wrote a few years ago, there are systematic differences between the attitudes of the public and of U.S. foreign policy elites:

Large majorities of Americans favor several specific steps to strengthen the UN, support Security Council intervention for peacekeeping and human rights, and favor working more within the UN even if it constrains U.S. actions. Large majorities also favor the Kyoto agreement on global warming, the International Criminal Court, the Comprehensive Nuclear Test Ban Treaty, and the new inspection agreement on biological weapons. Large majorities favor multilateral uses of U.S. troops for peacekeeping and humanitarian purposes, but majorities oppose most major unilateral engagements. . . . Analysis of more than one thousand survey questions asked of both the public and foreign policy officials over a thirty year period by the CCGA (formerly CCFR) indicates that significant disagreements between officials and the public have been very frequent . . . Over the years, however, there have also been many disagreements over Defense issues (the public is more reluctant to use troops and more opposed to military aid and arms sales), and even more disagreement on international Economic issues: citizens are more worried about immigration and drugs, and much more concerned about the effects of trade on Americans’ jobs and wages.

Perhaps most relevant to Saletan’s discussion above, Page concludes that this is more of a problem with the experts than with the public, concluding:

Most gaps between citizens and officials appear to have more to do with differing values and interests than with differing levels of information and expertise. To the extent that this is true and that Americans’ collective policy preferences are coherent and reflective of the best available information, there would seem to be a strong argument, based on democratic theory, that policy makers should pay more heed to the public’s wishes.

An interesting insight in Page’s paper is that policymakers may prefer unilateralism because they can envision themselves making the policy and would like more freedom of action. In contrast, citizens in general have a more distant perspective that might actually be more realistic—-given what we know from research in psychology about “the illusion of control.”

What Saletan calls “a bald admission that he’s willing to let public opinion override national interests” might better be characterized as a wise decision to go outside the unilateralist foreign policy consensus.

Continue Reading

PPP’s baffling discard process

B. J. Martino writes:

Earlier this summer when I went on a bit of a rant about PPP and their process of discarding interviews, rather than simply weighting data. Mark Blumenthal mentioned your response to the discusion in one of his posts, where you said you were a bit “baffled” by it.

While they claim to engage in the discard process as a kind of retroactive quota to account for more older, white women in their sample, it was the discards among the non-”older white women” that made me curious. That is, any respondent who was not meeting all criteria of being age 46+, white and female.

I downloaded the data from all their 2012 surveys for Daily Kos/SEIU, and compared the sample of non-”older white women” within the unweighted released data as well as the discarded data.

At least from the first six surveys I have looked at, there appears to be a consistent difference in the partisan composition of the released data and the discarded data for this group. In every case, the released data for this group was net Democratic in Party ID (Unw D-R), and the discarded data was net Republican (Dis D-R).

Party ID in PPP Polls for Daily Kos/SEIU- Non-”older white women”
(raw unweighted data and discarded data)




























































 

Unw Sample



Discarded Sample



Unw D-R



Dis D-R



Unw- Disc



25-Oct



968



335



6.5%



-1.5%



8.0%



12-Oct



1100



302



6.5%



-8.6%



15.1%



4-Oct



922



356



5.1%



-19.7%



24.8%



27-Sep



763



277



10.2%



-8.6%



18.8%



20-Sep



829



236



12.0%



-18.2%



30.2%



13-Sep



699



267



8.7%



-7.9%



16.6%



What this suggests to me is that the discard process is both a way to apply a retroactive quota to older white women, but also a way to fix the partisanship from another group (assuming this is primarily younger voters). My thought is that they are getting too Republican a sample in this group because they never dial cell phones.

I found it interesting that despite hanging their hat on being the most accurate of 2012, they announced today that they would be working to find a way to include cell phones in the future.


Martino continues:
As I told Mark, I’m not really interested into getting into a shouting match with PPP. It has always just kind of dumbfounded me how they work. The fact that Daily Kos/SEIU published all the raw data from PPP’s 2012 polling at least gave me some opportunity to figure it out.

I guess the troubling part is how they have repeatedly stood by the statement that they do not weight for Party ID, when this discard process would seem to indicate a de facto weight on Party ID for at least a portion of the sample. What they say is strictly true, but the effect is the same. Seems to be arguing semantics.

I also took a look at the Presidential ballot for this same group of non-“older white women.” Same effect, perhaps even a bit more pronounced.

Presidential Ballot in PPP Polls for Daily Kos/SEIU- Non-older white women
(raw unweighted data and discarded data)




























































 

Unw Sample



Discarded Sample



Unw O-R



Dis O-R



Unw- Disc



25-Oct



968



335



4.2%



-11.6%



15.8%



12-Oct



1100



302



-0.4%



-19.2%



19.6%



4-Oct



922



356



-0.4%



-30.4%



30.8%



27-Sep



763



277



10.3%



-11.2%



21.5%



20-Sep



829



236



9.7%



-22.1%



31.8%



13-Sep



699



267



6.3%



-15.0%



21.3%



 

I don’t really have anything to add here; it’s just an interesting story. I remain amazed that anyone would think it’s a good idea to throw away survey interviews that have already been conducted.

Continue Reading

Symposium Magazine September 2013 issue

College For All, Or Just For Some?
Judith Sebesta
All too often, Americans see a college degree as the ultimate insurance policy for success. But we need look to a far wider range of policy and educational tools to help those without a degree.

Why Democrats Are in Trouble in 2014
David C. W. Parker
Political reporting and punditry do a poor job in forecasting election results. Here is a closer look at how a political scientist would unpack the next elections.

Weather and War, Reconsidered
Scott K. Taylor
What the calamities of the seventeenth century can teach today’s scholars about climate change, war, and policy-making.

Never Mind the Generals, Here Come the Technocrats
Thomas E. Flores
Voters across the world increasingly prefer technocrats to run affairs. Why are they so popular?

Understanding the Irrational Commuter
David M. Levinson
The increasing sophistication of data collection and analysis gives us deeper insights into human behavior — and how we make decisions about everyday travel.

From the ACLU to Spy World to Academia
Chuck McCutcheon
Law professor Tim Edgar has worked on both sides of the surveillance debate, and he sees lessons to privacy advocates and government officials alike.

Fox, Meet Hedgehog
Euny Hong
A strategic studies program at Yale revives ancient lessons about statecraft, and its popularity is soaring.

Lots of good stuff here.

Continue Reading

Needed: peer review for scientific graphics

Under the heading, “Bad graph candidate,” Kevin Wright points to this article, writing:

Some of the figures use the same line type for two different series.

More egregious are the confidence intervals that are constant width instead of increasing in width into the future.

Indeed. What’s even more embarrassing is that these graphs appeared in an article in the magazine Significance, sponsored by the American Statistical Association and the Royal Statistical Society.

Perhaps every scientific journal could have a graphics editor whose job is to point out really horrible problems and require authors to make improvements.

The difficulty, as always, is that scientists write these articles for free and as a public service (publishing in Significance doesn’t pay, nor does it count as a publication in an academic record), so it might be difficult to get authors to fix their graphs. On the other hand, if an article is worth writing at all, it’s worth trying to convey conclusions clearly.

I’m not angry at the authors for publishing bad graphs—-scientists typically don’t get training in how to construct or evaluate graphical displays, indeed I’ve seen stuff just as bad in JASA and other top statistics journals—-but it would be good to catch this stuff before it gets out for public consumption.

Continue Reading

Here’s what happened when I finished my PhD thesis

PCAP_2013_Cover571

May 11, midday: Made a list of all the things I needed to do to finish the thesis.

May 12, midday: Tear up the list, decide to finish it with what I had at hand.

May 13, 7am: Thesis finished.

It was very satisfying.

P.S. This is the sort of post you won’t be seeing anymore, once we move to the Washington Post.

Continue Reading

Would today’s captains of industry be happier in a 1950s-style world?

In a post about “rich whiners,” Matthew Yglesias argues that what richies really want is respect. Yglesias writes:

I think rich businessmen would be happier if we could go back to 1950s-style, more egalitarian distribution of pre-tax income. The richest people around would still be the richest people around, and as the richest people around they would live in the nicest houses and drive the nicest cars and send their kids to the best schools and in other respects capture the vast majority of the concrete gains of being rich. But they’d also have a much better chance of gaining the kind of respect as civic and national leaders that they crave. They want to be seen as the “job creators” and the heroes of the economy, not the greedy exploiters of the masses. But in order to have heroes of the economy, you need a broadly happy story about the economy—one where living standards are rising across the board and prosperity is broadly shared.

This is an appealing argument but I’m skeptical. My impression is that, back in the 1950s, the culture heroes included sports starts, authors, broadcast and movie stars, etc. Some politicians and union leaders too, and various others. But nowadays, lots of rich people are heroes of one sort or another: Steve Jobs, those guys at Google, various gossip about tech billionaires, Donald Trump. Even the supervillians at Goldman Sachs get some respect—-it’s kinda cool to be a supervillian. There’s the Forbes list of billionaires. Not to mention Michael Bloomberg and Mitt Romney. And it’s not just cos these rich guys have done cool things like Google maps. Warren Buffett is a hero too, mostly from doing a very good job at accumulating money.

If you’re superrich and your goal is to be viewed as a hero, I’d say that the current era from the mid-90s through now has been a good time to do it. They call this the New Gilded Age for a reason. In contrast, I have the sense that the 1950s was a great time to get respect for local rich guys: the owner of the local factory, the proprietor of the local newspaper, etc. Back then, if you were running a moderate-sized business, you could be a real big shot.

I’m not quite sure how this could be studied more systematically but maybe it’s worth looking in to.

Continue Reading

“Why States No Longer Declare War”

Yesterday in this space Eric Grynaviski argued that “we [the United States] need declarations of war.” Given that Grynaviski is also suggesting that innocents inside of Syria be able to “exercise a veto over U.S. policy (if feasible),” I think it’s safe to say that his views are far from the mainstream of U.S. thinking. This is fine—it’s good for scholars to think outside the box—but I think it means that his view of “need” is pretty theoretical.

In any case, I thought it would be helpful to point to this paper (link to preprint here) from Tanisha Fazal, “Why States No Longer Declare War.” Fazal argues that “one set of norms—the rise of international humanitarian law—generates unintended consequences that include disincentives to comply with the long-held norm of declaring war.”

Continue Reading