“1.7%” ha ha ha

by Andrew Gelman on February 17, 2013 · 4 comments

in Methodology,Public opinion

Jordan Ellenberg writes:

Lots of people sharing this today.

Isn’t this exactly the kind of situation where they should have done some kind of shrinkage towards the national mean, as in that thing you wrote about kidney cancer rates by county? i.e. you see, just as you might expect, the extreme values of “proportion of people who said they were gay” are disproportionately taken by small states.

My reply:

If I don’t have the individual-level survey data that would allow me to do full-scale Mister P, yes, I’d fit a multilevel model to the state-level averages. I wouldn’t quite just partially pool toward the national mean; I think it would make sense to include some state-level predictors.

In any case, I think it’s tacky to report poll numbers to fractional percentage points. That kind of precision simply isn’t there.

{ 4 comments }

Nameless February 17, 2013 at 9:26 pm

It’s not obvious to me how one would even get a 1.7% out of 615 interviews. 10/615 is 1.63% and 11/615 is 1.79%. In either case, the sigma is +/-0.5% (on top of the sampling error), so, yes, the precision isn’t there.

Andrew Gelman February 17, 2013 at 10:46 pm

Nameless:

I assume the 1.7% is a weighted average. But in any case it’s ridiculous. But without the pseudoprecision they wouldn’t have as much of a story, hence not so many links (to which I contributed). The incentives go in the wrong direction.

Tobin Grant February 18, 2013 at 8:47 am

In grad school (late 90′s), our survey center did a poll on sexuality. Funny thing when we looked at the results on LGB (no T asked about at that time) — There was a jump in the number of bisexuals among those in their 70′s and older.

Why? The question was something like,

“Do you consider yourself homosexual or bisexual?”

The answer to most people was yes (pick a label) or no (heterosexual). But apparently older people in the sample saw it as a choice–which one are you? They weren’t aware of the term “bisexual”. They knew that they weren’t “homosexual” so they picked the other option.

Greg Lewis February 18, 2013 at 11:04 am

Although DC is an outlier in this survey, it’s also an outlier in a combined sample of full-time employees from the 2009-2011 American Community Surveys (2.3% of those in DC live with same-sex partners, compared to 1% in the next state (OR), which ties for #2 in the Gallup survey). Overall, the correlation between the ACS and Gallup measures is .81 with DC included and .50 without DC.

Comments on this entry are closed.

Previous post:

Next post: