The relevance of statisticians to researchers in different fields of social science

by Andrew Gelman on May 17, 2013 · 17 comments

in Methodology

Someone pointed me to a remark someone who felt that statisticians were not doing their job to help out “mathematically challenged” psychology researchers. My first thought was that statisticians often help in an indirect way, by developing advanced methods that quantitative psychologists then translate to their colleagues, but I also realized that there was some specific advice I could give that could be used right away. This made me think that my colleagues and I should put together a short document (an article? webpage? wiki? pamphlet?) of statistical advice. Maybe 50 useful tips. Much of this is in our books but it could be useful to have something that people can use right away, with no prerequisites and without feeling that it would be a big time commitment.

The thing I wanted to talk about here on the Monkey Cage, though, is that I can’t imagine a political scientist complaining that statisticians weren’t helping them out. The difference, I suppose, is that psychology has a longstanding field of psychometrics and mathematical psychology, and these people have been developing their own methods for many decades. In contrast, political methodologists take from other fields (mostly econometrics and statistics), so they are used to having to learn the language of others. And political scientists who are not methodologists (the equivalents of the “mathematically challenged psychologist” who asked the original question) know that if they want to do statistics, they have to learn some statistics; they don’t really expect to get by otherwise.

P.S. Just to be clear: None of this is meant to disparage qualitative research. Qualitative research is great. I’m certainly not saying that all researchers need to use statistics. This discussion is all about people who are doing quantitative work and fell the need to use methods that are somewhat beyond their mathematical/statistical comfort zone.

{ 17 comments }

Tracy Lightcap May 17, 2013 at 2:05 pm

Ok, I’m not a shrink, but I know a few. I think the problem here definitely isn’t a lack of maths.

All the psychologists I know are actually quite good at the kind of maths they use regularly. That means they are good at experimental design and inferential stats. They fancy themselves an experimental discipline these days and the rat-runners rule. They’re better at this then we are, easy.

Problem = a lot of the new data collection in the social sciences is based on Big Data; i.e. on amassing huge datasets and using descriptive stats to parse out what they are telling us. Here the psychologists I know will freely admit that they are pretty clueless and that they feel the lack strongly. They’re afraid – correctly – that they’ll be left at the gate as our understanding of human behavior increases. And, despite their leanings, they know that that is where their bread is buttered.

I feel their pain, but I don’t see an easy way out besides a large scale revival of social psychology. They tell me that’s sorta happening, but I don’t have time to keep up with what they’re up to.

Andrew Gelman May 17, 2013 at 3:05 pm

Tracy:

Just to be clear: the phrase “mathematically challenged” is not coming from me. I’m quoting the psychologist who started this discussion.

L Nettles May 17, 2013 at 2:51 pm

Steve McIntyre hasn’t gotten a lot of love for trying to help the statistically challenged.

John Mashey May 22, 2013 at 12:04 am

Just out of curiosity, for the statisticians here, if you found R code that had an explicit 1:100 cherry-pick to select the cases you wanted, and the result you claimed disappeared without that:
a) Incompetence?
OR
b) Falsification/fabrication?

And in either case, how much statistics advice would you take from the person who did it? (The code is explicit to sort, then sample from the top 100 of 10,000).

John McKellar May 22, 2013 at 5:08 am

I’d ask why you were cherry-picking; as that seems particularly relevant to the result you claimed.
So, neither fraufulent nor incompetence at this stage, just needing clarification of why these particular cases were selected.
Any conclusion should have the selection caveat included.

It sounds like the result only appears with the selection criteria – which could easily be a subset of particular interest and could be a finding that opens the door to further discoveries. On the other hand, it may be cherry-picking. ;-)

John Mashey May 22, 2013 at 4:49 pm

See Effect of selection… and the Deep Clinate post that shows the code and other problems.

The whole point of the original MM paper was to claim that a particular math technique generated hockey stick temperature curves. They generated 10,000 curves (using parameters with way-high persistence), sorted to put the 100 most positive hockey sticks at front of list, then sampled from them, with idea of showing that the blade of the Mann Bradleyy Hughes hockey stick was an artifact.
Read the 2 posts , then back to you: fraud or incompetence ?

John McKellar May 22, 2013 at 7:23 pm

I only see one post; the moyhu blog.
However, this is a much bigger can of worms than the original comment about helping psychologists! This appears to be climate change – is this an apropriate forum for discussion.

I do not regularly use the techniques described in the blog I could read, but I can see where it is going and it made interesting reading; just not this week please!

So catch me offline and I’ll willingly read around your question – but I wonder if other commentators to the blog are perhaps more familiar with the statistics.

John Mashey May 22, 2013 at 11:27 pm

Thanks, doing this from iPhone amidst hike.
The Moyhu post references a post at Deep Climate.
You might ask I.Nettles for his reason for injecting the comment in the first place. Indeed this has been thrashed to death elsewhere, and climate scientists often work with very good statisticians. The challenge, which actually is related to the generalized problem of mutually productive interactions of statisticians and other fields .

TC May 18, 2013 at 8:10 pm

“This made me think that my colleagues and I should put together a short document (an article? webpage? wiki? pamphlet?) of statistical advice. Maybe 50 useful tips. Much of this is in our books but it could be useful to have something that people can use right away, with no prerequisites and without feeling that it would be a big time commitment.”

Yes, please! That would be incredibly helpful to those of us who are just starting out!

John McKellar May 19, 2013 at 4:17 am

If statisticians write a one page guide on how to apply statistical principles, could a psychologist do a single page on resolving all of my emotional problems?

Better that we statisticians make ourselves more readily available to offer consultancy – and psychologists reach out for mathematical help.

TC May 19, 2013 at 11:28 am

John,
My colleague tried to hire a math professor as a consultant while she was finishing her dissertation, and he wanted to charge her $1,000 for what he said would take him 2-3 hours. Yes, perhaps her committee members should have helped her, but they didn’t, and she could not afford that fee. Most of us who are starting out cannot afford those fees.

Are you worried about giving your knowledge away for free? I understand that not everyone believes that sharing our skills in is part of our professional obligation to help fellow scholars improve the quality of our knowledge and discourse, ( though thankfully many do, and many have given their time and effort freely to help me and many others. I will do the same to pay it forward ). However, this is still an excellent choice when one is concerned about his own self interests. I have seen many successful business models in the academy and in the private sector that give a significant amount of information away freely, because they know that people who value the information will pay for additional help. In fact, as a small business owner for several years, I often employed the same method to gain new clients. But one important caveat is that the consultant services must be affordable enough for someone on an academic salary in the early years of their career.

Andrew Gelman May 19, 2013 at 9:11 pm

Tc:

I am happy to give away information for free. I am paid a salary so I don’t have to worry about supporting myself by keeping my knowledge scarce.

TC May 19, 2013 at 9:17 pm

Thank you Andrew Gelman. I was actually directing the comment to John. I very much appreciate your work and your willingness to share it with the rest of us.

John May 20, 2013 at 4:08 am

Two points – both really meant to be supportive.
Firstly:
One swallow doesn’t make a summer; and if a professor charged a lot once, it doesn’t mean that the next person will too. I felt there was an assumption that I charged high rates for my work. I have given free and cheap advice to charities and academics. Actually, I’ve given free advice to companies too.
I’m not salaried but believe that the gregarious approach brings in more business. The problems have usually been interesting and feedback has always been positive (both matter in work).

Secondly:
My post was to point out that a short set of notes was an inadequate replacement for professional experience in either statistics or psychology. While we all like the concept; it is lazy and seldom is sufficient. All researchers should be encouraged to interact with people from other disciplines and be willing to ask for advice.
I accept that I put this latter concept one directional; it most certainly works both ways.

dab May 21, 2013 at 3:18 pm

Professor Gelman:

I seem to recall you asking for suggestions about how best to use the Statistics Forum. I don’t have a well-formed idea, but this notion of “50 tips” seems like it could lead to one use. Maybe solicit short pieces comprising one or several tips from many statisticians and then compile (or get someone else to compile) them and anything interesting from the comments section into a draft 50 tips pamphlet/wiki? Just a thought….I know you’ve had trouble getting stories from people, but maybe statisticians will be more forthcoming with tips than with stories?

At any rate, I like the idea of such a document. It seems like there should be a way to crowd-source it….

John McKellar May 22, 2013 at 3:39 am

Here is a start (and maybe a finsh!), I took the concept of 50 tips and searched the internet on the basis that nothing is new.
Before delving in; I think all such short advice MUST be treated with caution and while I like TP Hutchison’s pages, I’ve not proof read them, just skimmed them.

Here is what I found from my first quick search:
http://www.scotland.gov.uk/Topics/Statistics/Browse/Health/scottish-health-survey/TipsBeforeDataAnalysis

The very first page I opened (the Scottish one, of course) made me realise the enormity of the suggestion – it raises more questions than it answers. This is helpful as we need to maintain our awareness that 50 tips is the equivalent of a degree bought over the internet! It also made me realise that consulting improves with age – the route to analysis is a complex path with many turns along the way. We should remember this as the 50 tips develops.

Then I came across TP Hutchison’s pages:
http://www.angelfire.com/biz/rumsby/ARES.html
http://www.angelfire.com/biz/rumsby/ASTUDY.html
These excite me as they are an attempt at the 50 tips idea.
I’m sure that the statistical reader will say this or that is not well developed but I’m also confident that the non-statistical reader will warm to the stepped approach and non-technical language.

It is a starting point.
I’ve written to Dr Hutchinson to acknowledge I’ve used his pages and ask if they were updated anywhere. The pages are 16 years old an he’s not listed on the staff at MacQuire (that I can see), but I’ll update here anything relevent that I find.

dab May 23, 2013 at 8:18 am

Thanks for these links. Following your advice, I did a little searching myself and came up with this link:

http://www.ma.utexas.edu/users/mks/statmistakes/StatisticsMistakes.html

It is a little more technical than the links you provided and so may not be appropriate to recommend to anyone who would self-describe as “mathematically challenged,” but I liked how the author broke things down into four categories:

1. Suggestions for teachers of statistics
2. Suggestions for consumers of research
3. Suggestions for researchers
4. Suggestions for referees of research articles and editors of journals

I also like the annotated references to further reading. Rather than a degree bought over the internet, I see the 50 tips idea more as an entry point that introduces people to important ideas in an accessible way, but also provides guidance about where more detail can be found. As Professor Gelman said, the tips are already in stats books. To me, the goal of the 50 tips would be to emphasize them (since many may be buried among lots of detail) and to gather them in one place (since they are probably scattered across many different books). I guess the best analogy is that I see this as being more like an extended review article on data analysis for non-statisticians. You don’t become an expert from reading a review article, but you get your foot in the door by getting the big picture and suggestions for further reading.

Comments on this entry are closed.

Previous post:

Next post: