Nancy Folbre is an economics professor at the University of Massachusetts, Amherst.
The Harvard economists Carmen M. Reinhart and Kenneth Rogoff ironically titled their much-celebrated book on financial instability and economic growth “This Time Is Different,” asserting that the last crisis was not unique. Rather, they contend it was the most recent manifestation of an age-old tendency for both governments and private investors to delude themselves about the dangers of debt.
Today’s Economist
Perspectives from expert contributors.
Doubly ironic, then, that Thomas Herndon, a graduate student in my department, was seeking to replicate an article that Professors Reinhart and Rogoff published in 2010 for an econometrics class when he discovered evidence of errors in their cross-national analysis of the impact of national debt on economic growth. Efforts to replicate influential research seldom get much attention, but this time was different.
It’s worth asking why. It’s also worth asking how the chance of similar errors might be minimized in the future.
I won’t recapitulate the details of the controversy, which received immediate attention on The New York Times site (in both Economix and in Paul Krugman’s column and blog) and in other news media, including The Washington Post, The Financial Times and National Public Radio.
Professors Rogoff and Reinhart have conceded that they overstated the negative effect of high levels of government debt on economic growth but insist that the error did not undermine their larger argument.
As much of this news coverage emphasized, the political stakes are high, because the paper in question has been frequently invoked as justification for austerity, in the concerted efforts to cut public spending both in this country and in Europe.
Journalists may also have been receptive because confidence in austerity policies has already begun to dissipate as a result of poor outcomes in Europe (a recurrent theme in Professor Krugman’s columns) and growing evidence of cruel consequences, like increased hunger among Greek children.
Further, journalists know the sound of a good story: a graduate student enrolled in a distinctly unconventional economic department in a financially pinched public university found an obvious mistake in research published in the leading journal of the profession by two prestigious Harvard faculty members. He joined forces with two faculty members, Michael Ash and Robert Pollin, to publish an online critique.
Because the mistake concerned calculation of some weighted averages in an Excel spreadsheet it could be more easily explained than a more technical statistical problem.
Less attention has been devoted to the timing of the revelation. It’s easier to describe the tiny bit of blood on Goliath’s forehead than to explain why it took almost three years for a pebble to reach that target.
It seems pretty clear that efforts to ensure that published results can be replicated have not been stringent enough. Although many important journals now require authors to make data sets and calculations available online, the May Papers and Proceedings issue of the American Economic Review, in which the original article was published, stipulates only that “papers are published only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication.”
Professors Rogoff and Reinhart, who drew some of their data for the article from publicly available sources and also made a larger data set available on the Web site for their aforementioned book, were not required to make the actual data or spreadsheet on which they based their specific calculations available to other researchers. Dean Baker of the Center for Economic Policy and Research complained publicly soon after the original article was published.
Mr. Herndon was able to obtain the now-famous spreadsheet only about two weeks ago, after e-mailing the authors repeatedly and explaining that he planned to publish a replication based on analysis of highly similar data yielding results significantly different from theirs.
Related difficulties have afflicted other economists seeking to replicate influential published articles. Setting rules is easy. To what extent are journal editors willing to actually monitor conformity and impose sanctions on those who fail to document their work in sufficient detail? Bruce McCullough of Drexel University, who has published extensively on these issues, suggested in an e-mail to me that editors have the power to prevent publication or to shame violators publicly.
On the other hand, detailed monitoring is time-consuming and costly. Austerity in both university and publisher budgets militates against it.
Further, as Dan Hamermesh of the University of Texas at Austin points out, the professional incentives to perform replications are weak and editors are seldom enthusiastic about publishing them. Indeed, replications that confirm previous results typically get less attention than those that controvert them.
As Professor Hamermesh puts it, “economists treat replication the way teenagers treat chastity – as an ideal to be professed but not to be practiced.”
This combination of weak incentives and weak norms weakens the reliability of economic research.
Still, the younger generation is often out front on replication because they relish a chance to cross swords with their elders. Significant example of controversies initiated by junior economists include Jesse Rothstein’s critique of Caroline Hoxby’s research on school choice and Justin McCrary’s critique of Steven Levitt’s research on electoral cycles in police hiring. Maybe graduate students should be even more actively encouraged to police the profession.
Replication problems aren’t unique to economics. Considerable evidence suggests that all scientists are prone to unconscious bias and subtle misperceptions and what are called “shoehorn effects,” trying to make results conform to expectations.
The most important remedy may lie in efforts to encourage the kinds of theoretical, political and cultural diversity that provide intrinsic motivation to challenge the conventional wisdom.
Two University of Chicago economists, George Stigler and Gary Becker, wrote a famous article entitled “De Gustibus Non Est Disputandum” – there’s no arguing over tastes.
In my department, our slogan is De Omnibus Est Disputandum: argue about everything.
Article source: http://economix.blogs.nytimes.com/2013/04/22/replicating-research-austerity-and-beyond/?partner=rss&emc=rss