How 2012 stacks up: The worst graph on record?

by Andrew Gelman on January 8, 2013 · 32 comments

in Environmental Politics,Methodology

Pat Egan sends along this monstrosity:

It would be a fun exercise for your methods class to find all the problems with the above plot.

I’m really glad to have it out there, though. Next time we mock Fox News for one of its ridiculous graphs, we can point to this even worse image for partisan balance.

P.S. They also supply a clean time series, lower down on the page:

Pretty scary, actually.

{ 32 comments }

MitchellCares January 9, 2013 at 6:05 am

Definitely a terrible graph by all means, but good look ever seeing this being ran on TV when there are much better graphs that describe the liberal viewpoint on climate change.

But one bad graph from one group does not equal partisan balance. A national TV news network running several bad graphs and misinforming its viewers vs one group with one bad graph. Strangely enough they had a similar graph but it was just related to March and it was a lot better than this one.

I know its a small point, but false partisan equivalence gets on my nerves.

ChristianKl January 11, 2013 at 3:12 pm

I would hold a group that claims it represents “leading scientists and journalists” like Climate Central to a higher standard in plot design than a TV newschannel. TV news don’t represent leading scientists.

Nate January 9, 2013 at 8:59 am

I feel like an idiot because I can’t find much wrong with it. Both axes are well marked and seem consistent with the actual posted values (in contrast to that infamous fox graph), it clearly states what measure is being displayed (contiguous U.S. average temperatures, by year) and what the source was (NOAA, which is as solid as sources come), and it clearly states that it’s displaying “the warmest years on record” rather than a time sequence. The y-axis starts and ends in logical places (consistent with the amount of variation seen in high temperature ranges, so we can see the difference between ’34 and ’06, for example), has no sneaky breaks or jumps, and is linear all the way through. I feel like the take-home point of the graph is transparent, and I know how to read it. Literally the only thing I see wrong with this graph is that I usually expect “#1″ in a rank-ordered series to be on the left instead of the right, but that’s almost more of a personal preference than some kind of mathematical principal. Help a guy out – what’s wrong with it?

Ashley May 23, 2013 at 4:27 pm

The x-axis should start at zero for a more accurate reading. By starting it out at 53 degrees the creator of the graph is being bias in the delivery. That’s what I see wrong with this picture.

John Nugent January 9, 2013 at 12:09 pm

Although it’s “clearly stated” that the graph shows the warmest years on record rather than a time series, I don’t think that’s how the average person will read it. It fails to give any inkling of the other 111 years in the time period 1895-2012, which the time series graph helpfully does (allowing readers to make up their own minds about what to conclude). Unanswered are questions like why the *six* warmest years were shown (as opposed to 3 or 25, etc.), how these six compare to other years, etc. The use of the 3-D effect is ill-advised and has the effect of exaggerating the size of the right-most bar. Get a ruler: at full size, the left-most bar is about 0.75 inches wide while the right-most bar is just over 1 inch wide. This yields a gap between the size of the effect in the data and the size of the effect shown in the chart, which Edward Tufte calls the “lie factor” in chapter 2 of his book The Visual Display of Quantitative Information, which contains numerous examples like this. He concludes, “To be truthful and revealing, data graphics must bear on the question at the heart of quantitative thinking: ‘Compared to what?’ The emaciated, data-thin design should always provoke suspicion, for graphics often lie by omission, leaving out data sufficient for comparisons.” (p. 74)

FreeDem January 9, 2013 at 12:15 pm

I’m with Nate on this one, it’s a good graph that shows exactly what it is saying it is going to show. Not only was 2012 the warmest on record, but it is a full degree warmer than the other top warmest years on record. Maybe the y-axis could be a larger range, but that would just make it harder to see the jump. Even the second graph you post has only a range from 50 to 55.5.

Andrew Gelman January 9, 2013 at 12:31 pm

Freedem:

I don’t think it’s a good thing that a graph “shows exactly what it is saying it is going to show.” I think it’s better for a graph to show some context. The lower graph in my opinion is much much better, as it shows the data in time context and does not use tricky camera angles to make the largest point appear even larger. (Commenter John above gives more details.)

Of course, I’m assuming a scientific or public-policy goal. To the extent that the goal is advertising or public relations, sure, then you might want to obscure any information other than your main point. But that sort of thing makes me uncomfortable.

For a non-statistical analogy, consider the use of a quotation in a news article or political ad. The best quotations, to me, show enough context that you can see where the speaker is coming from. But in an advertisement, you might extract just part of the quote to present a cleaner, even possibly misleading, story (ads for movies and attack ads against political candidates are notorious for this). I don’t like that sort of thing, even though I can see the motivation for doing it from the perspective of advertising or public relations.

Presenting information misleadingly in a good cause might, in balance, be a good thing (that’s probably what those Fox News producers thought too, when they prepared those notorious graphs), but it’s still misleading, and when both sides do it I think it degrades our discourse.

Andrew January 9, 2013 at 1:04 pm

Oh please. You’re saying this graph is just as bad as the Fox News whoppers because it doesn’t contain all the information you’d like? It’s a bad thing for a graph to show exactly what it says it shows? Even though every fact in the graph is 100% correct?

What world are you living in?

Don’t tell me it’s the (ahem) “tricky camera angles.” That’s clearly just post-hoc justification for your original lousy take. The perspective used in the graph is so slight as to be completely negligible. When they skew the bars but not the axes, then come talk to me about being as bad as Fox News. Until then, quit the phony false equivalence crap.

Andrew January 9, 2013 at 1:07 pm

Let me correct myself: You said this graph was WORSE than the classic Fox News howlers.

Just incredible.

Adam Hughes January 9, 2013 at 12:55 pm
Andrew January 9, 2013 at 12:57 pm

False equivalence FAIL.

This graph is completely honest, transparent, and is not misleading at all. Anyone with elementary-school-level reading comprehension skills can understand what it says, and what it says is true – 2012 was the hottest year on record, and by a significant margin. It purports to say nothing about trends or increase in temperature over time. It just shows 2012 in comparison to the previous hottest years.

So please, Mr. Gelman, do engage in your “fun exercise” and point out all the things wrong with this graph. We’re all waiting.

Andrew Gelman January 9, 2013 at 1:18 pm

Andrew:

See John’s comment above. The time series makes their point just fine with no selection or distortion required.

But, sure, I don’t really think it’s the worst graph on record; I was just riffing on the title of the original plot. The graph here, for example, is much worse, as it makes no sense even in its own framework.

And I think the equivalence to Fox News is true, not false. In either case you have an advocate who (I assume) feels strongly about their position, selecting and distorting data to make a cleaner point than would be obtained using a direct time series.

ChristianKl January 11, 2013 at 3:10 pm

- 3D Plots are misleading
- it’s deceiving to claim that you have 4 digits of accuracy for the years expect 2012. There no good reason for having only 3 digits of accuracy for 2012 but 4 for the previous years. 4 digits in 1921 seems wrong. You don’t have an infinitive number of measuring stations in the US but a finitive and based on those station you estimate the overall temperature. That estimation process doesn’t give you 4 digits of accuracy.
-the standard way to order years is to order them chronological
-53.0-55.5 is a qutie narrow range that’s choosen to exaggerate the effect.
-minor: are those mean or median temperatures?

Adam Stephanides January 9, 2013 at 2:42 pm

The graph is even more misleading than John says. If you look closely, the “horizontal” lines aren’t parallel, but fan out as you go from left to right, making the 2012 bar larger than it should be. When comparing the heights of the tops of the bars, the difference between 1998 and 2012 should be twice that of the difference between 1921 and 1998. Instead it’s at least three times as large.

As Andrew Gelman said, the straightforward time series is dramatic and scary enough, and so would a non-”enhanced” bar graph have been.

Lawrence Zigerell January 9, 2013 at 5:06 pm

Let’s say that the graph is improved by removing the three-point perspective, choosing a more justifiable baseline, adding a significant digit to the 2012 temperature, and extending the data to include more than six observations. The problem remains that the graph uses regional data to illustrate what is commonly proposed to be a global phenomenon when the global data are or will soon be readily available. This can lead to statements such as the earlier comments [plural] that describe 2012 as the “warmest” or “hottest year on record” without a contiguous-US-only caveat. Claims that the graph is not misleading might be more persuasive if the claims were not coupled with inaccurate statements about what the graph depicts.

Kevin Hill January 9, 2013 at 5:32 pm

These comments are a fascinating object lesson in how academics look at data differently than partisans.

Fabius Maximus January 11, 2013 at 5:08 pm

I agree with Hill. The technical details about the graph, the original intent of the post, are interesting. But more broadly it reminds us that there is no “neutral” presentation of information. One cannot show everythey relevant, and so authors show what’s most relevant for their purpose given constraints (eg, time, space, media, autidence).

This kind of teleology dominates discussions of public policy implications of cutting edge science. Cheerleaders (eg, non-professionals at WUWT, Skeptical Science) denounce as charlatans professional scientists that disagree with their “side” of the debate. Expert voices (eg, at RealClimate or Climate Etc) tend to get drowned out by the volume and exaggerated lanuage of the partisans.

We see this in the comments below about “cherry picking”. Both sides preference different time periods, using the trends that support their viewpoint. Today is a natural end point for a time series, the point where we live — and at which past become future . But what is the best starting point?

There is none. Each tells its own message. Temperature increases paused after aprox 1998. Anthropogenic forces began to dominate after aprox 1950. Warming started in the early 19th century. We can roll the starting point backwards, with periods of warming and cooling resulting.

It’s much like the different pictures of the world shown by different spacial scales in the films about “the Powers of Ten”. See their great website at
http://www.powersoften.com

Fabius Maximus January 9, 2013 at 6:03 pm

Although this graph certainly does not qualify as among the “worst” IMO,, it is in the Fox News Tradition of showing accurate data to produce a misleading context.

As Zigeral says, it is misleading to draw conclusions about global climate trends based on the 1.6 % of the Earth’s surface inthe 48 contgious States. Although the final data is still coming out from the various datasets, it’s clear that 2012′s global temperature was in the range of the past dozen or so years. Not scorching or broiling, as thenews media describe the US.

As such showing this by itself is accurate but misleading.

The latest forecast from the UK Met Office forecasts that this might continue for another five years (given the large uncertainty of these forecasts), based on their latest model. They also state, as have many climate-related organizations, that the cause of the current pause remains unclear — and that it is not unusual or conflicting with current climate model forecasts.

Andrew Gelman January 9, 2013 at 8:23 pm

I disagree with the claim that it’s inappropriate to look at U.S. data. I agree that global data are better, but (a) readers in the U.S. are interested in the U.S., and (b) the U.S. results are consistent with global patterns: according to the NYT article, “The year featured a La Niña weather pattern, which tends to cool the global climate over all, and scientists expect it to be the world’s eighth- or ninth-warmest year on record.”

Fabius Maximus January 9, 2013 at 10:31 pm

Andrew,

I broadly agree with you. US readers are interested in US. But let’s not feed our already exaggerated localism (exceptionalism, parochialism)!

The news media are working diligently to create alarm about global warming. Which is fine, except to the extent they exaggerate or create misleading impressions. Almost every major climate-related science institution has discussed the “pause” — now aprox 14 years long.

Instead they report almost exclusively data that reenforces their narrative. Unusual cold, snow cover, or low storm activity is “weather”; unusual heat, melting, or storms are “climate”. Unusual is often defined in terms of absurdly short baselines, as climate goes — such as tropical storm Sandy and the US drought — both called “historic” on flimsy foundations.

That’s the context for graphs like the one in this post.

This kind of activism is in effect a bet, with the news media remaining credibility as stake. It’s already alienated large elements on the Right (or rather, further alienated them). It’s probably a good bet, and they might never get called on it. Only losers have to explain.

But there are a real majority of climate scientists who disagree with the consensus forecasts, and have a significant body of peer-reviewed literature. They’re probably wrong, but minorities in science have been right in the past — and sometimes will be in the future.

Lawrence Zigerell January 10, 2013 at 3:32 am

Presenting US temperatures is not inappropriate or wrong or inaccurate or even misleading. But for the global phenomenon of global warming, US temperatures are a sample and global temperatures are the population, and readers of a graph of sample data might confuse patterns in the sample with patterns in the population. Such confusion is problematic to the extent that sample patterns and population patterns differ.

Sample patterns are consistent with population patterns in the second posted graph depicting mean annual temperatures from 1895 to 2012, so there is no problem using sample data in this context because the obvious implication of the graph is unchanged if the frame switches from the sample to the population: temperatures have generally been rising over the past few decades.

But if the graph provides information for only the six warmest years on record with 2012 being the rightmost and tallest column that is highlighted in a different color than the other columns and with the 2012 temperature label in a larger font size than the other labels, then sample patterns are not consistent with population patterns in that context, given that 2012 would not even be a data point on the equivalent “How 2012 Stacks Up” graph of population temperatures. In this case, the obvious implication of the graph — that it is hotter than it has ever been in the recorded data– is correct for the sample but incorrect for the population.

This is not a problem with the graph as much as it is a problem associated with the graph: multiple commentators described 2012 as the warmest or hottest year on record without differentiating between the sample and the population. I suspect that viewing the “How 2012 Stacks Up” graph would cause a nontrivial number of persons to think that 2012 was the hottest year on record across the globe, but in the absence of evidence I would at least propose the possibility of such a misperception as a potential problem for the hypothetical methods class to discuss.

Fabius Maximus January 10, 2013 at 12:08 pm

Thank you for the additional explanation! One important detail, however.

“temperatures have generally been rising over the past few decades.”

That only somewhat accurate, which is the key point. Global temperatures have risen in spurts for two centuries (since roughly 1950 primarily from antropogenic causes), so that the “past few decades” gives different pictures. Looking at US in 2012 is misleading in the sense that global temperatures have remained stable (no statistically significant change) since roughly 1998 (depending on the data set used).

It’s not a controversial finding, and has been discussed by the IPCC, UK Met Office, the Berkeley Earth Surface Temperature Project, many eminent climate scientists, a large body of peer-reviewed research, etc.

That it remains so little known might be, in part, from the use of graphs like that shown in this post — without giving a larger context.

If you’re interested, I can provide links to supporting sources.

kuiddler January 10, 2013 at 12:24 pm

Fabius. This is off topic so I’ll keep it short: if by “eminent climate scientists” at IPCC you include Edward Wegman, you should dig a bit deeper into the reliability of your sources. Start by searching Gelman’s site, keyword “plagiarism,” and following the links.

Fabius Maximus January 10, 2013 at 2:17 pm

“if by “eminent climate scientists” at IPCC you include Edward Wegman”

I don’t include him, for several reasons. To the best of my knowledge…
* he is not regarded as a climate scientist, and doesn’t claim to be such;
* he has never been “at the IPCC” or associated with them;
* to date he has not written about the “pause”.

He is a Professor of Statistics at GMU, and was disciplined because a paper prepared under his supervision used paraphrased material without proper attribution.

John Mashey January 10, 2013 at 8:27 pm

To be really precise, actually it was copy-and-paste mosaic plagiarism, of 1.5p of text with trivial edits, and GMU managed to ignore the 5pm superset of the same material in the Wegman Report, reported to them, as well as the 70+ other pages in other works.
They put a reprimand in his file, made him retract the paper (shoe retraction was forced by elsevier 10 months earlier, and apologize to the journal, ie the old colleague EiC who had accepted the paper in a few days w/o peer review.

A few months ago he was appointed to a 3-year term on the GMU College of Science Tenure ad Promotion Committee.

Fabius Maximus January 11, 2013 at 4:57 pm

Not to interrupt the jihad, but I think the only relevant part of this is that kuiddler’s inplication that Wegman was one of my sources is false, as was kuiddler’s description of Wegman.

Lawrence Zigerell January 10, 2013 at 1:10 pm

Hi Fabius,

Thank you for the always-welcome offer of supporting sources, but I have come across reports of the lack of statistically significant global warming since 1998. I should therefore have been more careful in my language and used a phrase such as “since 1960 or 1970″ instead of a phrase that can reasonably be interpreted as “over [each of] the past few decades.”

jrkrideau January 10, 2013 at 2:41 pm

Perhaps the key point is ‘statistically significant” .

If you cherry pick 1998 then there is apparently no statistically sigificant warming. I believe if you extend the series even as far back as 1997 it is statistically significant.

A rather smarmy but interesting film clip covers this http://www.youtube.com/watch?feature=player_embedded&v=u_0JZRIHFtk

Millsy January 10, 2013 at 10:29 am

As I sit here with my masters level intro stats Lecture #2 on the basics of graphs, I must thank you for this one.

John Mashey January 10, 2013 at 2:15 pm

1) Year-to-year variations from ENSO (and big volcanoes, when they happen) are much bigger than yearly trends, and the statistics of such time series *demand* that such graphs seem to have spurts. Far more consistent is the growth in Ocean Heat Content.
Watch the Escalator, as used by Senator Whitehouse and recent PBS/Frontline.

2) 1998 was the biggest El Nino ever recorded, and starting with that is a cherry-pick. See #7 on list of common wrong arguments. See this at Open Mind, a blog run by a good time-series guy.

3) Indeed, the first graph was awful, the second is better. Even better is to look at analysis like this, which uses historical data to remove the natural fluctuatiosn so you can see the trends.

4) Wegman has never been associated with the IPCC.

John Nugent January 10, 2013 at 4:48 pm
Gena July 30, 2013 at 8:34 pm

Thank you for the auspicious writeup. It in reality was once a amusement account it.
Look advanced to far delivered agreeable from you!
However, how could we communicate?

Comments on this entry are closed.

Previous post:

Next post: