Archive | Health Care

Is coffee a killer? The statistical significance filter strikes again

coffee Thomas Lumley writes:

The Herald has a story about hazards of coffee. The picture caption says
Men who drink more than four cups a day are 56 per cent more likely to die.

which is obviously not true: deaths, as we’ve observed before, are fixed at one per customer.  The story says
It’s not that people are dying at a rapid rate. But men who drink more than four cups a day are 56 per cent more likely to die and women have double the chance compared with moderate drinkers, according to the The University of Queensland and the University of South Carolina study.

What the study actually reported was rates of death: over an average of 17 years, men who drink more than four cups a day died at about a 21% higher rate, with little evidence of any difference in men.  After they considered only men and women under 55 (which they don’t say was something they had planned to do), and attempted to control for a whole bunch of other factors, the rate increase went to 56% for men, but with a huge amount of uncertainty. Here are their graphs showing the estimate and uncertainty for people under 55 (top panel) and over 55 (bottom panel) FPO-1 There’s no suggestion of an increase in people over 55, and a lot of uncertainty in people under 55 about how death rates differed by coffee consumption. In this sort of situation you should ask what else is already known.  This can’t have been the first study to look at death rates for different levels of coffee consumption. Looking at the PubMed research database, one of the first hits is a recent meta-analysis that puts together all the results they could find on this topic.  They report
This meta-analysis provides quantitative evidence that coffee intake is inversely related to all cause and, probably, CVD mortality.

That is, averaging across all 23 studies, death rates were lower in people who drank more coffee, both men and women. It’s just possible that there’s an adverse effect only at very high doses, but the new study isn’t very convincing, because even at lower doses it doesn’t show the decrease in risk that the accumulated data show. So. The new coffee study has lots of uncertainty. We don’t know how many other ways they tried to chop up the data before they split it at age 55 — because they don’t say. Neither their article nor the press release gave any real information about past research, which turns out to disagree fairly strongly.

I agree.  Beyond all this is the ubiquitous “Type M error” problem, also known as the statistical significance filter:  By choosing to look at statistically significant results (i.e., those that are at least 2 standard errors from zero) we’re automatically biasing upward the estimated magnitudes of any comparisons.  So, yeah, I don’t believe that number. I’d also like to pick on this quote from the linked news article:
“It could be the coffee, but it could just as easily be things that heavy coffee drinkers do,” says The University of Queensland’s Dr Carl Lavie. “We have no way of knowing the cause and effect.”

But it’s not just that.  In addition, we have no good reason to believe this correlation exists in the general population. Also this:
Senior investigator Steven Blair of the University of South Carolina says it is significant the results do not show an association between coffee consumption and people older than 55. It is also important that death from cardiovascular disease is not a factor, he says.

Drawing such conclusions based on a comparison not being statistically significant, that’s a no-no too.  On the plus side, it says “the statistics have been adjusted to remove the impact of smoking.”  I hope they did a good job with that adjustment.  Smoking is the elephant in the room.  If you don’t adjust carefully for smoking and its interactions, you can pollute all the other estimates in your study. Let me conclude by saying that I’m not trying to pick on this particular study.  These are general problems.  It’s just helpful to consider them in the context of specific examples.  There are really two things going on here.  First, due to issues of selection, confounding, etc., the observed pattern might not be real.  Second, even if it is real, the two-step process of first checking for statistical significance, then taking the unadjusted point estimate at face value, has big problems because it leads to consistent overestimation of effect sizes.

I’m posting this here (as well as on our statistics blog) because I think these points are relevant for political science research as well.

Continue Reading

Autism and the social contagion of information

The news that Jenny McCarthy will become a co-host of TV discussion show The View is generating a lot of controversy – people worry that McCarthy will be able to spread her controversial (for which read: crazy) views on autism, vaccination and chelation therapy to a much wider audience. Some of the theories as to what causes autism point to geographic clustering as evidence for some common physical cause. However, Columbia sociologist Peter Bearman and his collaborators build on a rich body of data from California to show that clustering of autism cases is plausibly caused by increased diagnosis thanks to the diffusion of knowledge across local communities. Ka-Yuet Liu, Marissa King, and Bearman find that at least 16% of the recent increase in diagnoses is down to the spread of knowledge:

One does not “catch” autism from someone else, yet a social diffusion process contributes significantly to the increased prevalence of autism. We observe a strong positive effect of proximity to other children with autism on the subsequent chance of diagnosis, robust to a range of individual- and community-level controls in both urban and less urban areas. In addition, close proximity to a child with autism was inversely associated with the likelihood of subsequent sole MR diagnosis, while it correlated strongly with the chance of autism-MR diagnosis. Proximity also increases the chance of autism rather [than] MR diagnosis given the same level of severity in autism symptoms. Social influence arises strongly for high-functioning cases of autism. The effect of proximity is also more prominent in younger children, when diagnosis is more difficult and parental resources are more important. Children who were diagnosed with autism have a similar mode of referral as that of their nearest neighbor with autism before their diagnosis. All of these findings are consistent with a mechanism of social diffusion of awareness of the symptoms and the benefits of treatment and are inconsistent with competing explanations. Social influence also accounts for the observed spatial clustering of autism. Such clustering could be caused by local environmental toxicants, the diffusion of a virus, or residential selection, but it is hard to see how a toxicant could cause a reduction in MR diagnoses, operate in all types of communities (urban or rural), and affect most strongly the high-functioning end of the severity distribution.

It’s likely that the increase in diagnoses reflects the difficulties in getting assistance within California (and the US education system more generally) for children who urgently need help, and who would be denied it without a full autism diagnosis:

Because the DDS provides services only to children with autism and not to children diagnosed with disorders on the autism spectrum, the importance of an autism diagnosis for parents striving to secure resources for their children is amplified. The steep and sudden cliff creates incentives that may not be present in other contexts, but pressure to do anything to help children is likely widespread and not limited to the California context. As Judith Rapoport of the National Institute of Mental Health told Grinker, “I’ll call a kid a zebra if it will get him the educational services I think he needs.”

Anecdotal evidence suggests that parents of kids with autism or related issues face grueling battles if they want local schools to acknowledge their needs and provide for them. It wouldn’t surprise me at all if much of the knowledge that is shared locally is knowledge of how to force action from a bureaucratic system where administrators have limited budgets and strong incentives to deny care if they think they can get away with it.

Update: This shorter piece by Bearman on why many parents believe that vaccines cause autism is also interesting.

Continue Reading

Fixing America’s Physician Shortage with Immigrants

Skills are often occupation-specific, a fact missing from existing research on the political economy of immigration. Although analyses of survey data suggest broad support for skilled migration occupational licensing regulations persist as formidable barriers to skilled migrants’ labor market entry. Regulations ostensibly serve the public interest by certifying competence but are simultaneously rent-preserving entry barriers. We analyze both the sources of US states’ licensure requirements for international medical graduates (IMGs), and the effect of these regulations on migrant physicians’ choice of US state in which to work over the period 1973-2010. Analysis of original data shows that states with self-financing state medical licensing boards, which can more easily be captured by incumbent physicians, have more stringent IMG licensure requirements. Additionally, we find that states that require IMGs to complete longer periods of supervised training receive fewer migrants. Our empirical results are robust to controls for states’ physician labor market. This research identifies an overlooked dimension of international economic integration: implicit barriers to the cross-national mobility of human capital, and the public policy implications of such barriers.

That’s from a new paper from Brendan Peterson, Sonal Pandya, and David Leblang.  The (catchy) title is “Doctors With Borders: Occupational Licensing as an Implicit Barrier to High Skill Migration.”  A press release is here, with this quote from Pandya:

…unelected U.S. state regulators are dictating the international transferability of skills through occupational licensing barriers, and removing such barriers has the potential to address some of our most pressing public policy concerns.
Continue Reading

Do Americans Trust Goverment Less Because It’s Become an Insurance Broker?

At the end of an interesting and data-packed post on the growth in federal spending, Nate Silver offers a hypothesis:

Nevertheless, the declining level of trust in government since the 1970s is a fairly close mirror for the growth in spending on social insurance as a share of the gross domestic product and of overall government expenditures. We may have gone from conceiving of government as an entity that builds roads, dams and airports, provides shared services like schooling, policing and national parks, and wages wars, into the world’s largest insurance broker.
Most of us don’t much care for our insurance broker.

Nate suggests that the growth in spending on social insurance is changing what goverment does and perhaps making people trust the government less.  The former is undoubtedly true.  I am more skeptical about the latter.  Here is why.

Nate looks at the growth in entitlements as a percent of GDP, and categorizes spending on health care, education, and welfare as entitlements.  (The data are available here.)  I think I’ve adequately replicated the trend he documents, though focusing on the period from 1958-2011.

And now the trend in trust in the federal government.  This is the percentage who say that you can trust the “government in Washington” “just about always” or “most of the time,” using data from the American National Election Studies, supplemented with an October 2006 CBS News poll.  (Adding more data from public polls would show the same trend.)

The two trends don’t appear to square that well.  Entitlement spending increases consistently, if not linearly.  But trust in government has important peaks and valleys.  It hasn’t simply declined since the 1970s.

What happens if we plot the relationship for those years were we have both sets of data?

Using all the data, 1958-2008, there is a negative correlation: more entitlement spending coincides with lower trust in government.  But this relationship leans heavily on the period from 1958-1972 (or 1958-1974 or 1958-1976—you can cut the data a couple different ways and get the same finding).  Once we focus on the period from 1974 onward, there is no meaningful correlation.  This is despite the fact that entitlement spending increased from 9.5% of GDP in 1974 to 15.7% in 2008.  So a lot depends on how you want to interpret the decline from 1964-1972 or thereabouts.  Entitlement spending did increase during this time—it nearly doubled—but there were also an economic slowdown, the Vietnam War, and Watergate.

Of course, these data don’t speak to the more subtle question of whether popular conceptions of government have changed as government has devoted more resources to entitlements.  That said, it’s also worth mentioning that entitlement programs like Social Security and Medicare are among the most popular (though less so Medicaid and welfare, obviously).  People may not care for their insurance broker, but they do seem to like government insurance.

[Full disclosure: I have been an occasional contributor to the 538 blog.]

Continue Reading

Death Panels and the Challenges of Correcting Political Misperceptions

Context: Misperceptions are a major problem in debates about health care reform and other controversial health issues.

Methods: We conducted an experiment to determine if more aggressive media fact-checking could correct the false belief that the Affordable Care Act would create “death panels.” Participants from an opt-in Internet panel were randomly assigned to either a control group in which they read an article on Sarah Palin’s claims about “death panels” or an intervention group in which the article also contained corrective information refuting Palin.

Findings: The correction reduced belief in death panels and strong opposition to the reform bill among those who view Palin unfavorably and those who view her favorably but have low political knowledge. However, it backfired among politically knowledgeable Palin supporters, who were more likely to believe in death panels and to strongly oppose reform if they received the correction.

Conclusions: These results underscore the difficulty of reducing misperceptions about health care reform among individuals with the motivation and sophistication to reject corrective information.

That is the abstract of a new paper by Brendan Nyhan, Jason Reifler, and Peter Ubel.  A gated copy is here.  A neat graph is at Brendan’s blog.  It’s consistent with Nyhan and Reifler’s other research—e.g., here—that shows that correct information can backfire, especially among the people least likely to believe it in the first place.

Continue Reading

The Effect of the Court’s Ruling in the ACA

The Court’s perceived stamp of approval for the ACA led some Americans to switch their minds about it, leading to a small increase in approval of the law following the decision. Some of this opinion transformation might simply have come from the favorable media attention heaped on the law as a “winner” at the Court. In other words, the Court’s upholding of the law sent a signal that it was objectively good policy. For still others, the decision provided an opportunity for elite discussion and persuasion, instituted in particular by the President, who either convinced them on the law’s merits or triggered their latent approval for him by expressing new-found support for the law. In other words, precisely because the decision clearly defined the political stakes and discussion surrounding it became more politically polarized, support for the Act (which had lagged presidential approval) now became
more closely correlated with it.

From a new paper (pdf) by Andrea Campbell and Nathaniel Persily.  Of course, amidst a presidential campaign, this subject might feel like old news.  But really understanding the impact of the Court’s ruling can’t be accomplished within a news cycle.  This paper is a nice, and still rather timely, review of the evidence.

Continue Reading

Will the Recommendation Against Ovarian Cancer Screenings be Embraced? Understanding Public Skepticism About Research-Based Treatment Guidelines

This is a guest post from University of Virginia political scientist Eric Patashnik.


The United States Preventative Services Task Force on Monday issued a report advising healthy women who do not have symptoms against receiving routine screening tests for ovarian cancer. According to the expert panel, the screenings are not effective in lowering the death rate from the disease, but generate a high percentage of false-positive results that “can lead to important harms, including major surgical interventions in women who do not have cancer.” This is not the first time that a common diagnostic test for cancer has been questioned. This task force had earlier advised against prostate specific antigen (PSA) tests for healthy men and against annual mammograms for healthy women under the age of 50. Many medical experts argue that American patients are overdiagnosed and overtreated, and that evidence-based guidelines can improve the quality of health care.

It remains to be seen whether the recommendation on ovarian cancer screenings will be embraced by physicians, patients, and policymakers. The mammogram guideline was of course very controversial. Many patient groups expressed outrage about the perceived emphasis on reducing health care costs, and some Republican officeholders said it illustrated the kind of rationing that will occur under the Affordable Care Act. A Pew survey found that 68 percent of people who followed the news about the change in the mammogram guideline disagreed with the task force’s recommendation. The Obama Administration, already fighting allegations that it favored “death panels,” responded to the outcry by promising that government insurance programs would continue to cover mammograms for women starting at age 40.

As the U.S. struggles to control health care costs, it is important to understand how the American public thinks about these issues. What arguments about evidence-based treatment guidelines does the public find most and least convincing? In 2009, Alan Gerber (Yale University), David Doherty (Loyola University Chicago) and Conor Dowling (University of Mississippi) and I performed a national survey of 1,026 respondents to investigate this question.

Our survey found some public support for the main arguments that experts give for using research-based guidelines. Nearly two-thirds of survey respondents were somewhat or very convinced that economic incentives might cause doctors to give patients unnecessary care or the wrong care. In addition, a little more than half were somewhat or very convinced that doctors can lose touch with the latest research and that they follow local standards of care and therefore may be unaware of better treatments being used elsewhere. However, no more than 20 percent of respondents found these arguments to be very convincing.

In general, respondents found the arguments against guidelines more convincing. More than eight in ten were convinced that guidelines would prevent doctors from tailoring care to the needs of individual patients and that no outside group should come between doctors and patients in making treatment decisions. More than seven in ten believed that guidelines would be used to ration care, are vulnerable to abuse, and would not keep pace with medical innovation. The proportion of respondents finding each of these arguments to be very convincing ranged from 29 percent to 51 percent. Among subgroups, Republicans and seniors were most wary about guidelines.

Proponents of evidence-based medicine will need to address these concerns before treatment guidelines can win broad acceptance among ordinary Americans.

Continue Reading

“If our product is harmful . . . we’ll stop making it.”

I’m used to hearing the argument that, sure, cigarettes are addictive, but everyone has known forever that smoking caused cancer, and that cigarette manufacturers could hardly be blamed for supplying a consumer good that many people wanted. So I was surprised to learn the following, from historian Robert Proctor:

It’s interesting to see that, at least in public, cigarette executives taking a much more direct position that they did not want to be in the position of giving people cancer: “If our product is harmful . . . we’ll stop making it.”

Further background (including the cartoon of Fred Flintstone smoking) at the sister blog.

Continue Reading

Have the Republicans Fought to a Draw on Medicare?

Republicans say yes.  Steven Benen agrees.  I think that conclusion is premature.  The Democrats and Obama retain key advantages on this issue—that is, this particular fight is not yet a draw.  The Republicans continue to fight an uphill battle.  Here is why.

1) Do people know what Ryan’s Medicare plan is?

Most do not.  In an August 18-20 YouGov poll, respondents were asked this question:

Republican vice presidential candidate Paul Ryan has proposed a plan to change Medicare, the federal government’s health insurance program for older Americans. Which of the following most accurately describes Ryan’s proposal for Medicare?

– Gradually end the Medicare program over the next 40 years – Turn Medicare into a government voucher program for the purchase of private insurance – Transfer control of the Medicare program from the federal government to the states – Have not heard about Ryan’s plan

In this sample, 40% picked the correct answer (voucher program).  The largest group (43%) said that they had not heard.  Democrats were most likely to pick the correct answer (45% did), followed by Republicans (38%) and pure independents (36%).  The majority of pure independents (58%) had not heard of the plan.

2) Among those who do know Ryan’s plan, is it popular?

Among those who picked voucher program, 58% oppose it and 34% favor it.  The intensity of opinion favors opponents: 53% strongly oppose it.  Naturally, Democrats are more likely than Republicans to oppose it, and Democrats’ opinions are more intense.  Among independents, 60% oppose it (with the caveat that this figure is based on a small sample—only 60 independents).  This NY Times poll and this Kaiser/Washington Post poll also found opposition to Ryan’s plan.

3) Which party does the public trust to handle Medicare?

As I noted in my first post on this subject, Democrats traditionally have an advantage on Medicare. The August YouGov poll confirmed this.  About 34% of the sample trusted Democrats while 27% trusted Republicans (the remainder said both or neither party).  I can put this in the context of other issues.  The Republicans have a small advantage (2 points) on the economy and a larger advantage (7 points) on the budget deficit.  The Democrats have a small advantage on foreign policy (2 points) and a larger advantage (8 points) on Social Security.

Most importantly, the Democratic advantage on this hasn’t moved much in the last year—despite the recent campaigning.  My first post cited a couple earlier polls that produced results consistent with this YouGov poll.

4) Which candidate does the public trust to handle Medicare?

Consider the gap not between the Democratic and Republican parties, but between Obama and Romney.  Here’s a list of findings from August polls that asked which candidate was better on Medicare:

CNN: Obama +5 (registered voters) and Obama +1 (likely voters)

Washington Post: Obama +1 (adults) and Romney +3 (registered voters)

AP/GfK: Obama +8 (adults) and +6 (registered voters)

Fox: Obama/Biden +7 (registered voters) and +3 (likely voters)

In addition, some battleground state polls give Obama an edge of +8 to +10 points among likely voters.  (Seniors tend to trust Romney more than Obama.  Of course, they also tend to support Romney over Obama, period.  More below on how seniors do—or do not—respond to Medicare campaign messages.)

In fact, Obama’s advantage on this issue isn’t that much different than the Democratic party’s edge—at least among registered voters and adults.  It is a bit smaller in the national polls of likely voters but not polls, but not the 3 state polls among likely voters.

5) Can the GOP talk successfully about this issue?

Have the Republicans found a successful frame for talking about a traditionally Democratic issue?  In my earlier post, I noted that parties often “trespass” on the other party’s issues by framing that issue to fit their strengths.  The GOP seeks to do that by tying Medicare to Obamacare, which is obviously far less popular than Medicare.  Here is Jay Cost advocating that strategy.  Here is an NRCC case study from a Nevada special election. Here is a slide with a poll question from Resurgent Republic, showing a Democratic argument and this Republican frame fighting  to a draw:


The question, of course, is whether any survey question like this is an adequate test of campaign messages.  It doesn’t capture all the claims in this Obama ad, for example.  But I don’t meant to criticize Resurgent Republic.  The bigger problem with message testing in any survey is this: it’s like testing someone’s reaction to a blizzard by holding an ice cube against their skin.  A swirling blinding mess is distilled to a clear identifiable cue.

A lot of campaigning boils down to what I would call “frames vs. fundamentals.”  When that is the case, bet on the fundamentals.  The fundamentals here are these: Medicare is an issue that still favors the Democrats and Obama, and a majority of Americans oppose Ryan’s plan for Medicare—both when they are able to state what it is and also when it is described for them in a poll.  This issue is not yet a “draw.”  It’s still Democratic turf.

For framing to work, the GOP must get voters’ attention even as many tune out the campaign.  The GOP must make its particular frame stand out amidst the blizzard.  That is difficult.  And even if the GOP can get voters’ attention, the frame has got to get some of them to change their opinions—also a difficult task, as much social science attests.

Ultimately, the Republicans also have to think about trade-offs.  Given all of the possible issues they could discuss, is Medicare the best use of their time?  Is it a better issue than the economy or the deficit?  Where will they get the most bang for their buck?

To be sure, in the case of Medicare, the Obama campaign clearly has some work to do too, mainly because many people do not know what Ryan’s plan is.  But the Romney campaign arguably has to do more.

6) Will any of this Medicare debate actually mobilize seniors to vote?

Let me end on a cautionary note.  It is easy to imagine that campaign advertising about Medicare will send seniors streaming to the polls to defend Medicare from the vicious cuts of Paul Ryan or Barack Obama or whomever.  But when Andrew Karch and I studied Medicare advertising and senior turnout in the 1998, 2000, and 2002 elections, this is what we found:

…we found no consistent evidence that messages related to Social Security and Medicare were associated with higher turnout among seniors

Which is to say, regardless of whether and how you think the parties should engage Medicare, it might not matter in the end.

[Cross-posted at Model Politics]

Continue Reading

On trust fund “raids” and “double counting”

This is a guest post by University of Virginia political scientist Eric Patashnik.  Earlier posts by him are here and here.


The big Medicare fight between Obama and Romney/ Ryan is over whether to retain or alter the program’s traditional defined-benefit structure.  That’s where political attention properly belongs.  But there is also an interesting side skirmish over whether Obama has “raided” the Medicare Hospital Insurance Trust Fund to pay for the coverage expansions in the Affordable Care Act.

It is a powerful, morally-charged accusation, because it suggests that President Obama “broke faith” with the seniors who paid into the Medicare system during their working lives.  Trust-funding financing clearly does not eliminate political conflicts over taxes and benefits; rather, it shapes what the conflicts are about.

Romney’s accusation goes back to earlier GOP charges of “double counting” – the claim that the Obama Administration fudged the numbers when it stated that the Medicare spending cuts contained in the ACA both lower the federal budget deficit and extend the solvency of the HI Trust Fund to 2024 from 2016.

In fact, as former CBO Director Robert  Reischauer points out, the Obama Administration followed the budget accounting rules that both parties have used for decades.  According to the Center on Budget and Policy Priorities:

the “Balanced Budget Act of 1997 and the Deficit Reduction Act of 2005 — both of which Republican Congresses approved — included Medicare savings that were counted as reducing the deficit and improving Medicare’s financial outlook.  The Senate Republican Policy Committee rightly claimed credit for this result, and no one made charges of double-counting.”

This is not to say that the federal government’s budget rules are perfect. Many experts believe that Washington’s budgeting procedures and accounting conventions lack transparency and consistency, and fail to promote fiscal responsibility. There is no shortage of proposalsfor reform. Some proposals would require the president and OMB to use the CBO budget baseline.

More radical are calls for the federal government to adopt accrual or generational accounting. If federal budget officials used different accounting conventions than they do today, the numbers might change some, but the bottom-line conclusions about the nation’s fiscal situation would remain the same:  our taxes are low by international standards  (as Andrea Campbell points out, we spend too much on health care and the nation is on an unsustainable fiscal path).

Medicare is funded in part from a contributory trust fund because President Johnson, Wilbur Mills and other program architects wanted to give workers “earned rights” and promote fiscal rectitude.  Those were hard goals to merge when Medicare was created back in 1965, and they are even harder to reconcile today.

Continue Reading