In “Race and the Moral Character of the American Experience,” Paul Sniderman and Edward Stiglitz marshal data from a new national survey (this one conducted via internet by Knowledge Networks) to speak anew to some enduring and extremely controversial issues concerning Americans’ racial attitudes. This, in essence, is the N-th round (where N is a very large number) in what has at times flared into a heated controversy concerning how full the glass of racism really is. A distinctive new element here, though, is that rather than focusing single-mindedly on prejudice against African Americans, Sniderman and Stiglitz reasonably argue that researchers and others should also consider the offsetting force of good will toward African Americans, and they go on to do exactly that.
Everything I’ve ever read by Paul Sniderman (and I’ve read just about all of his published work, which is voluminous enough to have kept me quite busy over the years), is a model of lucidity. His books and articles are “good reads” that lay out the argument forcefully and bring data to bear on the argument, often in innovative ways. By the same token, my main reservations about his work are that at times the arguments seem to me to have been laid out too forcefully and that the data don’t seem to me to support the argument so clearly. Perhaps unsurprisingly, those are also my main reservations about “Race and the Moral Character of the American Experience.”
Before going any further, I want to acknowledge that I take the Sniderman-Stiglitz article to be an early report on a project about which we’ll be hearing more in the future. It’s not yet a finished product, and some of my concerns may well be allayed in future papers elaborating their ideas and presenting their results in greater depth and detail.
In any event, I have three main concerns about Sniderman and Stiglitz’s arguments and evidence. Each of these concerns begins with a methodological point, but each methodological point leads to a substantive one. These points are as follows: (1) I’m less willing to dismiss as inconsequential threats to the validity of the data that Sniderman and Stiglitz present than they appear to be. That being the case, I suspect that they understate the degree of prejudice against African Americans and overstate the degree of good will. (2) Much more importantly, I think Sniderman and Stiglitz define prejudice in an unduly restrictive way. If that’s so, then the effect is again to understate prejudice against African Americans. (3) Finally, I question whether the comparison of absolute levels of prejudice against, and good will toward, African Americans that Sniderman and Stiglitz present is appropriate. This skepticism leaves me unpersuaded that the latter really does offset the former.
I develop these points below the fold.
(1) It is well known, and abundant prior research demonstrates, that when survey respondents are asked about “touchy” subjects (and sometimes when the subjects aren’t even touchy), they have a tendency to answer in ways that cast themselves in a positive light. This is hardly surprising, but it puts a large proportion of findings based on survey research under a cloud. For example, there’s the well known phenomenon of voter turnout being higher when survey respondents say whether they voted than it actually was on Election Day; voting is supposed to be a good thing, and many survey respondents report having done it even when they didn’t.
The same sort of “social desirability” effects haunt research on racial attitudes. The so-called “Bradley effect,” which got a lot of publicity during the recent presidential campaign, is one example. It may or may not have operated to any substantial degree during the campaign, but the general phenomenon that it represents has often been documented.
I wouldn’t single out the Sniderman and Stiglitz study for special criticism in this respect (indeed, their results are less susceptible to social desirability-based responding than those reported in many other studies) except for their two-sentence dismissal of this problem:
bq. “First, the interviews were conducted over the Internet. Respondents thus had no need to doctor their responses to present a socially desirable image to an interviewer.”
Taken literally, this statement must be true. In an internet survey, there is no interviewer. Thus, there is no need for survey respondents to present a socially desirable image to an interviewer.
What that statement does not say, though, is that there is no need for respondents in an internet survey to present a socially desirable image, period. Face-to-face contact with an interviewer does seem to exacerbate the social desirability problem, so the absence of an interviewer obviously helps avoid the problem. However, it doesn’t follow that this problem can be wholly discounted in analyzing data from an internet survey. It can’t be. Some people some of the time answer sensitive survey questions in ways that tell us more about themselves as they would like to be – their idealized selves – than as they really are, no matter whether or not there is someone sitting right in front of them asking the questions. One reason for this tendency is to avoid projecting a negative image of oneself to the “generalized other.” Another reason, certainly no less fundamental and perhaps more so, is the tendency to avoid projecting a negative image of oneself to oneself.
Accordingly, I’m not prepared – or at least as prepared as Sniderman and Stiglitz appear to be – to take at face value the survey responses upon which their analyses are based. Yes, eliciting these responses in ways that are more anonymous and less threatening is less likely to bring out socially desirable responding than would be the case in face-to-face or even telephone interviews. But the problem doesn’t simply go away.
Why belabor this point? Because Sniderman and Stiglitz want, among other things, to compare the levels of negative and positive affect toward African Americans that the survey respondent express (on which, more below). To the extent that socially desirable responding is in play, respondents will understate their negative affect toward blacks and overstate their positive affect. And any such tendency gets directly in the way of interpreting the results as Sniderman and Stiglitz do. That is, if one’s argument accentuates the positive and downplays the negative, then survey results that seem to support that argument become less persuasive when it is taken into consideration that these results reflect – perhaps in substantial ways – social desirability-based biases in responding to survey questions.
(2) Much more importantly, some of Sniderman and Stiglitz’s key interpretations and conclusions seem to me to say more about their working definition of prejudice than about prejudice per se. Here is Sniderman and Stiglitz’s definition of racial prejudice:
bq. “By racial prejudice we mean a consistent, systematic, predictable tendency to respond strongly and negatively to black Americans.”
This definition sets the bar very high. An occasional tendency to respond strongly and negatively to black Americans wouldn’t qualify as prejudice. A tendency that seems out of character for a certain individual – a “blind spot” – wouldn’t qualify, either. Neither would a less-than-strong tendency.
If one employs a very restrictive definition of prejudice, then one is likely to get a low count of prejudiced people.
To see how this plays out in Sniderman and Stiglitz’s results, consider their analysis of the incidence of prejudice. Survey respondents were asked whether each of five negative adjectives (“violent,” “boastful,” “complaining,” “lazy,” and “irresponsible”) describes most blacks “extremely well,” “very well,” “moderately well,” “slightly well,” or “not at all.” They assign scores of 1.0, .75, .50, .25, and 0.0, respectively, to these responses. For each respondent, they then calculate a mean score, which obviously can vary between 1.0 (for someone who answered that all five of the adjectives describe most blacks “extremely well”) down to 0.0 (for someone who answered that all five describe most blacks “not at all”). “Any cutting point is arbitrary,” Sniderman and Stiglitz concede, “but defining a score of .7 or more as high on this Prejudice Index is not unreasonable.”
Really? Imagine a respondent who has answered “moderately well” on all five items. That is, s/he thinks “violent,” “boastful,” “complaining,” “lazy,” and “irresponsible” are all pretty good descriptions of blacks. That respondent, with a Prejudice Index score of .50, wouldn’t qualify as racially prejudiced by Sniderman and Stiglitz’s standards. Or how about someone who thinks that the first three of those uncomplimentary adjectives describe most blacks “very well” and the last two “moderately well”? That person sounds pretty darned prejudiced to me, but even his/her score on the Prejudice Index (.65) would fall below the very high bar that Sniderman and Stiglitz have set. Of course, Sniderman and Stiglitz can set their cutting point anywhere they please, but I take with a sizable dosage of salt the conclusion that they draw – that 10 percent of the population is racially prejudiced. That’s what I mean when I say that their conclusion says more about their cutting point than it does about racial prejudice per se.
Once again, this methodological point has major substantive implications. After considering the data I have just described, Sniderman and Stiglitz conclude: “By common sense standards, then, a nontrivial proportion are still systematically hostile to blacks, even though the largest proportion of the general population are not.” This conclusion depends for its validity on Sniderman and Stiglitz’s restrictive measurement decisions. For whatever it’s worth, if (following one of my examples above) the bar were set at .50 on the Prejudice Index, then it would still be true that “the largest proportion of the general population are not “systematically hostile to blacks,” but I think the headline of this story would be that one-third of the general population is systematically hostile to blacks.
(3) What Sniderman and Stiglitz consider the novel contribution of their analysis is their incorporation of good will (“esteem”) toward African Americans into the mix of racial attitudes. I am persuaded that this is an idea worth pursuing, but I am also persuaded that it needs to be pursued very carefully.
Here is the statement that sets off alarm bells for me:
bq. “The mean score is higher on the Esteem Index than on the Prejudice Index …”
Yes, it is. But are absolute scores on the two indices comparable? Imagine that you and I are teaching separate sections of an introductory American government course. On my mid-term exam, the mean score is 86 and on yours it’s 72. Should we conclude that my students performed better than yours did? Well, we should draw that conclusion if we know, among other things, that the students in the two classes took the same test under the same conditions. But, just to take one possibility, suppose that my test was full of genuinely easy questions and yours was full of genuinely hard ones. Then the differential in scores between the two classes could well reflect no more than the differential degree of difficulty of the two tests.
The point of this little exercise in psychometrics is that Sniderman and Stiglitz’s comparison of prejudice and esteem scores depends on the problematic assumption that the items on the two scales are equally “difficult” – that the scale items are semantically balanced. Are they? Well, maybe and maybe not — that’s an empirical question. I don’t know whether the items were calibrated carefully, or whether they were calibrated at all, because no evidence about the psychometric properties of the two sets of items is presented in the article. That’s not a criticism, for this is closer to a working paper than to a full-form, technical presentation. However, before I could accept the conclusion that esteem is more common than prejudice, I’d have to know considerably more than this article tells me. Until I’m persuaded that there is a sound psychometric basis for comparing the scores on the two scales, I’m going to have to put that attempted comparison on hold.
Overall, then, my reservations about some of the key conceptual and measurement aspects of this study undermine my confidence in the findings that Sniderman and Stiglitz report and the conclusions that they draw. I hasten to add that further analysis may reveal that these findings are indeed trustworthy and these conclusions warranted. For now, though, I view them as inviting dismissal for being methodological artifacts. That would be a shame, for the issues that are raised in this article are fundamentally important ones.