Both left wing and right wing commentators complain frequently about the rise of journalistic ‘factchecking’ columns arguing that they substitute personal (and perhaps biased) opinions for purportedly neutral evaluations of politicians’ factual claims. A more sophisticated version of the critique suggests that factcheckers face their own sets of incentives – i.e. for example, that they may need to appear ‘balanced’ by criticizing both left and right reasonably evenly, even if one side (depending on your viewpoint) is systematically more mendacious than the other. Which makes this “defense of factchecking”:http://mediapolicy.newamerica.net/sites/newamerica.net/files/policydocs/The_Rise_of_Political_Fact-checking.pdf.pdf by the founder of the Washington Post‘s factchecking column, Michael Dobbs, particularly interesting.
What is striking about Dobbs’ account is the apparent inconsistency in its defense of ‘Pinocchios’ – the pseudo-quantitative score that the factchecking column awards (from one to four Pinocchios) to dishonest political statements (the more Pinocchios, the more dishonest the statement is supposed to be). On the one hand, Dobbs suggests that the Pinocchios are opinions, not dicta:
bq. While I agree that reporters should not be partisan, I see no reason why we should not draw on our experience and plain commonsense to sort out the truth from fiction. Reporters should be allowed to sift the evidence and reach conclusions. I see nothing wrong with that, as long as the conclusions are based on the evidence, rather than reflecting some kneejerk political position. A Pinocchio is just that, a conclusion, expressed in a colorful way. It is the evidence-based opinion of a reporter who has examined the facts, and tried to determine the truth as best he can. Readers, and politicians, are free to agree or disagree.
On the other, he and his successor think that the Pinocchios are what get the attention, and seem pleased with that attention.
bq. Kessler defends the Pinocchio device as a way to whet the interest of readers and attract the attention of the candidates, some of whom are “obsessed with Pinocchios.” He has received calls from senior politicians wanting to know what they need to do in order to avoid receiving more Pinocchios. “I have learned my lesson,” said one errant member of the
House of Representatives. “I am not going to utter another word on this subject without having my staff check it over.” … While they are quick to ridicule fact checks aimed
against them, campaigns often gleefully jump on fact checks aimed at their opponents. “Four, count ‘em, Four Pinocchios,” the Obama campaign crowed when I exposed Hillary Clinton’s Walter Mittyish fantasy about “coming under fire” at Tuzla airport while on a visit to Bosnia. I knew that “Pinocchios” had entered the political lexicon when I tuned into a Republican presidential debate in late 2007 and heard Mitt Romney berate one of his Republican rivals: “You earned three Pinocchios for that.” (He did not feel the need
to explain exactly what a ‘Pinocchio’ was.)
So, on the one hand, Dobbs is telling us that Pinocchios are just an expression of a specific journalist’s considered and evidence based opinion, which readers and politicians are free to agree or disagree with. On the other, he seems to be suggesting that it’s great that so many people take them so seriously. Something there seems to me not to compute. This is more worrying still when you throw in Dobbs’ admission that he sometimes felt out of his depth in writing the column.
bq. I often felt overwhelmed by the deluge of misinformation on the campaign trail. While I was well versed in some topics, such as foreign affairs and history, I was a neophyte on others, including the complexities of the federal budget and the environment. In order to adjudicate factual disputes, I had to become an instant expert on a wide range of issues. There were times when I felt I was the only reporter researching the policy proposals of the candidates, rather than their day-to-day maneuvering and their standing in the opinion polls. Producing my unofficial quota of one fact check a day was sometimes a struggle. Looking back, I feel that I did more distinguished work during the 2004 campaign, when I was able to focus on a few issues, such as the Swift Boat controversy, than in 2008, when I was spread very thinly.
This suggests that Pinocchios are likely to be a pretty unreliable measure of anything except for obvious lies or truths. When a single person is trying to make judgment calls on very different topics every day, most of which he or she has no subject-matter expertise on and then is expected to score these judgment calls using a semi-arbitrary ranking scheme, the final product is likely to have a pretty low information content. Not only that, but it is likely to have a particularly pernicious mix of apparent authoritativeness and actual unreliability. Readers are likely to pay a lot of attention to it, because it looks quantitative and definitive, without thinking much about how it was arrived at. It might be interesting to see media scholars doing experiments, in which they varied the number of Pinocchios awarded as their treatment, while keeping the verbal description of assumed dishonesty fixed, to see exactly how much the quantitative frame shaped individual perceptions.
Dobbs’ suggestion (if I read him correctly) that the factchecking column is a poor man’s substitute for substantive journalism on the content of policy is also very interesting. However, that’s a different topic for a different post (most likely on a different blog).