Dave Armstrong writes:
I [Armstrong] recently written a blog post that might be of interest to the Monkey Cage readers concerning the recently released Heritage Foundation Index of Economic Freedom—specifically what they did wrong methodologically and how they could fix it with some original analysis of their data and re-examination of their key conclusions.
Armstrong writes:
Recently, the Heritage Foundation released its 2010 Index of Economic Freedom. No doubt, in the interest of replicability and transparently, Heritage released all of the data required to produce all of their indices. . . . Heritage employs a simple method of using multiple measurements to get a better sense of what is happening with economic freedoms around the world. Their main conclusions are that Hong Kong and Singapore still top the list of most economically free countries and that the US is rapidly and significantly losing economic freedom as it falls out of the seven top “free” economies.
But then:
I [Armstrong] argue that these findings don’t tell the whole story. When a more appropriate model is used to estimate economic freedom, Denmark and New Zealand are found to be the two most free economies and the US, while significantly less free than these top two economies, is not significantly different from the any of the other top 20 economies. While I find that the US does have a lower score this year than last, it is not a statistically significant drop (i.e., from a statistical point of view, economic freedom in the US is not different in 2010 than it was in 2009). In fact, none of the top 20 most free economies is significantly more or less free in 2010 than they were in 2009.
I haven’t read Armstrong’s blog in detail, nor have I seen the Heritage Foundation Index before, but I like the idea of taking this sort of index apart and understanding how it works. Those of you who are interested in this topic can follow up from here.
{ 9 comments }
Can someone, maybe Armstrong, explain to the mathematically challenged, why Armstrong removed both the “Government Spending” and “Fiscal Freedom” variables? I’ve criticized the index before for including both variables, which I think measure much the same thing twice, just from different perspectives. And I’ve noted that some countries (like Denmark) get heavily dinged on the fiscal variables despite doing extremely well on all the others. But tax burden is intuitively an element of economic freedom. So what’s the idea?
Thanks for your question, Will. Let me say a couple of things. Measurement is a process that has both deductive and inductive stages. The initial stage is deductive in that we choose a set of variables that would all seem to be indicators of the same underlying variable (e.g., economic freedom). You can think of this as positing a hypothesis that all of the variables chosen are reliable indicators of the concept of interest. This is where Heritage stops — they posit that the 10 variables chosen are all equally good indicators of the same underlying concept (economic freedom) and they proceed under the assumption that they’re right without ever testing the hypothesis. While this often seems like a reasonable idea in the measurement world, we would generally not let such a thing pass in other corners of social science.
What I did was test their hypothesis about the reliability of these indicators. What we’re both (Heritage and myself) trying to do is estimate the latent trait of economic freedom. To illustrate my point, imagine that we already know exactly what economic freedom is – that is to say, we have one variable that perfectly measures economic freedom. If Heritage is right about the reliability of their 10 chosen variables, then each of the 10 variables would be well predicted by this perfectly measured economic freedom variable. The technique I used looks at all of the observed variables (the 10 indicators) and it tries to come up with the an estimate of this latent trait that predicts all of the observed variables as well as possible. The idea is that if we can find one thing that predicts all of these variables well, that thing is likely to be “economic freedom”. What I found was that while eight of the variables were well-predicted by this one underlying variable, two of these variables, Government Spending and Fiscal Freedom, were not, they were essentially unrelated to the underlying trait. This serves as a test of the hypothesis originally (and implicitly) posited by Heritage that all 10 variables are reliable indicators of economic freedom. An investigation of the data confirmed the hypothesis for eight of the variables and rejected the hypothesis for the remaining two.
The take away point is this – the (seemingly reasonable) intuition that tax burden is an indicator of economic freedom is useful in the deductive stage of the measurement process – in choosing a set of variables that *should* be indicators of the latent trait. However, it is only through an investigation of the data that we can evaluate the appropriateness of that hypothesis and generate an appropriate measure of the latent trait.
Dave, Thanks very much. I think I get it. How about this as an emendation of the index… Let’s say that what you’ve discovered is that the Heritage conception of economic freedom has two dimensions. One dimension is captured by the index after discarding the Government Spending and Fiscal Freedom variables. The other dimension is captured by the discarded variables (and maybe others that have been overlooked). So we can say that a country can be economically free in one way without being economically free in another. Some countries, like Singapore and Hong Kong, are very free in both ways. Others, like Denmark, are very free in one way but not in another. Which is better depends on how much one values each kind of economic freedom.
Similarly, suppose we wanted to measure economic equality, and found that equality in income and equality in wealth often come apart. I don’t think we’d want to throw one of the variables out as not really tracking economic equality. We’d say they both track a different kind or different dimension of economic equality. No?
Dave – I’m happy to see people take apart the Heritage index – which is problematic on a whole range of issues – but I disagree with you that a latent variable approach to measurement is the only accepted practice in the social sciences.
In fact, there is a long tradition in political science, associated with the work of Sartori, and more recently Collier and even more recently Gary Goertz that has taken a very different approach to the psychometric tradition of thinking of concepts as latent variables.
And so I think a deductive approach as you call it is perfectly acceptable in a large chunk of the social sciences (to wits – none of the commonly used indices of democracy – including the more ‘academic’ Polity IV use a latent variable approach).
Intuitively that also makes sense – we may well think of different factors of “economic freedom” as imperfect substitutes – e.g. a harder time opening a business can be partially compensated by having to pay less taxes on it. The nice thing of these type of concepts is that you know exactly what goes in there and how.
What I think is wrong with the Heritag index is a “garbage in – garbage out” problem: Dennis Quinn and co-authors have a paper where they make minced meat of the Heritage measurements of capital account restrictions (part of investment freedom).
Transparency international data for corruption is highly questionable (Markus Kurtz has an article on that).
Some of their scoring rules are completely arbitrary – e.g. gov’t freedom is
100-0.03*(Gov’t expenditure)^2 – in order to punish higher gov’t spending more strongly,
the financial freedom is measured in pretty losely defined and coded steps of 10, which are then assumed to be cardinal(!) values…
I could go on.
Will, I completely agree. As you rightly suggest, the other way of looking at this is -if these 10 variables represent what Heritage thinks is important to the concept, how many dimensions are needed to model it effectively. In this case, it appears as though 2 is the right number. The resulting 2 dimensions might be related to each other or not; that is something that could be built into the model, but the idea that the concept may be multidimensional is one that could be pursued just as easily.
Sebastian, thanks for your post. I both agree and disagree with parts of what you’ve said. Certainly your concerns about the indicators deserve careful consideration. The models I discussed generally assume that the measures are error-laden indicators of the underlying concept, so they permit a certain about of “garbage”, though less is generally better.
I also agree that there are lots of measures used regularly that do not explicitly appeal to psychometric theory. The methods discussed in my post are generally intended to reduce measurement error and they find the coefficients and latent variable scores that optimally do that for the defined problem. I could amend my previous statements to say that the only “statistical” justification for adding up variables in an effort to reduce measurement error comes from an appeal to psychometric theory and latent variable estimation.
As for Polity, Shawn Treier and Simon Jackman have a nice piece in the American Journal of Political Science that discussed Polity as a latent variable using methods quite similar to those used in my post. Another similar indicator with an ad hoc approach to measurement is Freedom House’s Freedom in the World. I have a working paper that uses similar techniques to show that there is actually a lot more information in the Freedom House data than the organization would have us believe.
thanks for your reply, Dave.
As for errors – obviously model based approaches allow for error – but only when the error is not systematic (i.e. in this case not related to the other factors).
That’s not the case for CPI scores (Kurtz shows that perception scores are sensitive to overall economic performance, for example and so their error is probably related to a whole bunch of the other factors), it’s – by design(!) not the case for their scoring of “government freedom” and I’m pretty sure if one would take a closer look it would also not be the case for a number of the other indicators – as a rule of thumb I tend to believe that when you can quickly see something is obviously wrong with a measurement that’s likely to be systematic even in a relatively narrow context.
As for the psychometric approach – sure adding more variables is the only justification if you’re interested in an underlying latent variable . But that’s not what all measurement is and I think part of the “quants” that want to sell this as the only statistically valid approach to measurement are ignoring a large literature on measurement.
For example, if you believe that certain components are necessary for a concept (as e.g. Pzeworski et al. do in their democracy index) a latent variable approach is completely misplaced.
Similarly, as I’ve stated above, if you think (as Heritage may well do – though I’m not sure) that different aspects of economic freedom are complements even if they are not related a latent variable approach is misplaced.
As Munk and Verkuilen’s 2003 piece on indices of democracy points out, what matters is that the measurement/aggregation strategy is in line with the conceptualization itself – there is, contrary to what you seem to suggest – not one right way of doing this. Rather, what matters is consistency.
I have trouble seeing “economic freedom” as a latent variable. In my mind, a latent variable is one that *causes* variation in the indicators. Because the effects are more or less predictable, we can infer something about the level of the latent from the levels of the observed.
But I can’t conceive of economic freedom as *causing* indicators like Government Spending or Trade Freedom to vary. It’s more like an *effect* of having relatively little of the one and a lot of the other, or maybe just a label for such a condition. But not a cause.
Now if we were talking about “commitment to economic freedom” (fuzzy as that is) instead of economic freedom itself, I’d be willing to treat that as a real thing that acts causally on the indicators, and then a latent variable approach would seem more reasonable.
Morgan: You seem to be referring to the distinction between what Ken Bollen called “effect indicators” (the latent variable influences the observed indicators) and “causal indicators” (the observed indicators influence the latent variable).
I have no problem considering both as models for latent variables. However, it is good to distinguish between the two, as both require very different strategies for recovering the latent variable.
Bollen, K. A. 1984. Multiple indicators: Internal consistency or no necessary relationship? Quality & Quantity 18(4): 399–385.
Bollen, K. A., and R. Lennox. 1991. Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin 110(2): 305–314.
Comments on this entry are closed.