Counterintuitive Findings, Revised Version

by John Sides on November 23, 2008 · 9 comments

in Political science

Matthew Yglesias linked to my post below, and his commenters got me thinking about ways to improve the “analysis.” (I put that it quotes to denote the fact that this is a cheeky blog post, not a “study,” as commenter “JimboSlice” referred to it.)

First, upon revisiting the work of my research assistant, I discovered that he had not in fact searched for the phrase “counterintuitive finding” but for the word “counterintutive” appearing in the article simultaneously with the word “finding.” This isn’t quite the same thing, obviously, so I searched for the phrase “counterintuitive finding” or “counterintuitive result,” allowing either “counterintuitive” or “counter-intuitive.”

I then searched for “intuitive finding” or “intuitive result.” This latter phrase will count as something of a control, accounting for the general increase in the number of journals—a point raised to me by Lee in conversation and by a couple of Yglesias’ commenters.

I also focus on publications through the 1990s, since since the 2000s aren’t over and the JSTOR database isn’t up to date on those articles that have been published in this decade.

Here’s the graph:

counterintuitive2.png

This confirms what I suggested below: there is an increase in the propensity of published research to describe its findings as “counterintuitive.” This does not appear to be an artifact of the sheer increase in journals or journal size, or otherwise the phrase “intuitive finding” would manifest a similar increase. Between the 1970s and 1990s, the use of the phrase “intuitive finding” increased by a factor of 4.5. The use of the phrase “counterintuitive finding” increased by a factor of 24.

I will leave it to the reader to judge whether I am still, as Yglesias commenter “Anonymo” put it, “full of warm gaseous vapors.”

{ 9 comments }

Andrew November 23, 2008 at 5:49 pm

John,

Nice graph. I have nothing to add; I’m just commenting here so you won’t think that my comments on graphs are always critical.

kyle November 23, 2008 at 8:26 pm

most likely reason: people don’t feel compelled to state the obvious, as is the case with intuitive results

Ben November 23, 2008 at 10:31 pm

I’m not sure if “intuitive finding” is really much of a control, even if it is the clearest linguistic opposite of “counterintuitive finding”. This would assume that academics always use the most obvious antonym and are not affected by whatever words have the most currency at a given time, etc.

But I do struggle to find examples of other phrases they might use… how many papers have ever admitted drawing “obvious conclusions”?

John Sides November 24, 2008 at 6:57 am

Andy: I’m speechless!

Kyle: It’s certainly true that papers don’t want to admit obvious conclusions, but the interesting question to me is why that isn’t (apparently) constant over time. Why the increase in “counterintuitive results”?

Ben: I’m open to other suggestions. I can try “obvious conclusion” if I have time.

Ben Sirolly November 24, 2008 at 8:53 am

In my experience, many science papers focus on counterintuitive observations. As Asimov said, “The most exciting phrase to hear in science, the one that hearlds new discoveries, is not ‘Eureka!’ but ‘That’s funny…’”

This might be evidence for the humanities becoming more scientific and observational. At least, I hope so.

In any case, great graph!

Doug H November 24, 2008 at 1:19 pm

My new pet phrase is to declare a finding: “non-trivial.” Maybe more clever phrasings can be found at the Journal of Irreprocible Reults? See:
http://www.jir.com/

Matt Jarvis November 24, 2008 at 2:18 pm

I’m kinda with Ben on this: I’m not sure I’ve ever written the phrase “intuitive finding.” I think that part of the problem is that there are a lot of different ways we say we “got intuitive results,” but “counterintuitive findings” has become a bit of a catch-phrase. I blame formal modeling for that. (I also blame formal modeling for the death of disco, but all I have there is a correlation).

Jeff Lazarus November 24, 2008 at 3:44 pm

FWIW, I think what’s going on here is an increased pressure on political scientists to “sell” their work.

lylebot November 24, 2008 at 11:20 pm

There’s no reason whatsoever to think that all words and phrases scale at the same rate. As the number of publications grows, the frequency of some phrases will grow at a faster rate than the frequency of others. This has been observed again and again in every language that exists.

Comments on this entry are closed.

Previous post:

Next post: