Numbers Game

A book that attacks habits ingrained in a whole profession, implicitly threatens several academic and civil service reputations, and names a few names is likely to arouse passions. The reviewer should then state his bias and interest. I have read this attack on the use of statistics with tranquility, indeed with Schadenfreude. An anxious glance reveals that my name is not in the index, my speciality (Sovietology) but lightly touched upon. Indeed this is no accident: the Sovietologist is by hard experience wary of all statistics; he customarily footnotes each single one, shows his working, records his inaccuracies and makes samokritika by publicly revising his mistakes. Mr. Morgenstern has little, though not nothing, to teach him. Indeed he could also learn.

The thesis is very simple: economic statistics are much more inaccurate than economists admit, and do not bear much of the heavy superstructure of assertion, prediction, and theory erected upon them. The thesis is demonstrated in regard to international trade (country A’s exports to B never correspond to B’s reported imports from A, the discrepancy reaches immense heights); price indices (a price is an extremely complicated notion if you take into account varying qualities, kickbacks, delivery dates etc.); and agricultural output (the Bureau of the Census disagrees with the Department of Agriculture, and the latter makes the most shameless revisions of old figures). A longer chapter, and a good one, takes us round the maze of employment and unemployment; the different data from different sources, the impossibility as yet of international comparison, the impossibility of defining unemployment, etc.

Most of the time the author’s touch is sure. In particular he does not overplay his hand, and allows that differences of definition and other legitimate causes may be more important than error in explaining discrepancies. With a dramatic thesis of this kind, Mr. Morgenstern is indeed to be congratulated on keeping his head; his book will have all the more ultimate influence. Perhaps rightly, too, he keeps off statistical theory. His eye is solely upon the accuracy of statistical observation, and he hammers this single point home well.

But in one chapter the author’s error is surely greater than his victims.’ It is quite true as he claims, that sheer estimates of the national income in a given year embody gross error, of the order of 10 per cent after ironing out all differences of definition. But it surely does not follow at all that in a normal estimate of the growth of national income these errors can be added. Thus if the national income in year 1 is really 100, and the estimate made is 92, Morgenstern will have it that in year 2 the truth might be 110 but the estimate 119. I.e., we might easily move from an 8 per cent underestimate to an 8 per cent overestimate, obtaining growth of 32 per cent instead of 10 per cent. It seems to me most improbable that the error in the second year would differ very much in direction or magnitude from that in the first. For the sources of error are neither random nor fluctuating, as the author suggests, but rather stable. I hold, therefore, that growth estimates, and change estimates generally, are not less reliable—as the author would have it—but more reliable than sheer estimates at a point of time; e.g., if in the above example our first estimate was 92 our second would in all probability be 101-103. Moreover this optimistic notion is not merely a priori. Empirical arguments could be urged, did space permit. The failure to see this possibility is a most serious omission, since the author’s excessive skepticism about growth statistics gives a handle to those who would have the U. S. Government do nothing about growth. Indeed Fortune has already grasped this handle (November 1963).

The example illustrates a general point. It is not enough to show that data are not reliable; the author has not taken the next, essential, step: to ask exactly what the effects of this unreliability are. Bad data are like a ladder with rotten rungs. You can’t walk gaily up the middle carrying a hod of bricks, but you can still walk gingerly up, holding tight to the shafts and putting your feet on the ends of the rungs. You can also lean against the ladder, and use it to measure heights. It is in a word “the best ladder you’ve got,” and such is the true art of the economic statistician. One misses this whole philosophy here: it is as if the author were not a user but only a critic of statistics, who would rather destroy than repair. Not every theory collapses because its statistical base is shaky. The figures have to be shown to be inadequate for the purpose for which they are used.

All criticism aside, however, the book performs an immensely valuable service. It should be required reading among all users of business and economic data. Let us hope, though, that it is not seized upon to justify governmental inaction (“After all the figures don’t mean anything, do they, old man? Hasn’t that chap Morgenstern shown we don’t know what’s going on anyhow, so probably everything’s all right anyway?”); nor retreat into abstract model-building (“A meaningful interpretation of expression (19.07) would require that the critical parameter be estimated to three decimal places; however Morgenstern (1963, p. xxx) shows that current estimates of Δ have $$$ 48%; we shall therefore pass this question by, and proceed to deduce expression (19.08) as follows:”); nor a renascence of metaphysical woolliness (“In this textbook I have not followed the current fashion of overloading the text with so-called statistical data. The erroneous nature of most of those currently in use has recently been shown by Mr. Morgenstern (ref.). I had rather let the basic principles speak for themselves”). To the author’s great credit, none of these false inferences is even hinted at. His objectivity is admirable.

The true lessons of his work are surely these. First, the data must simply be improved. It is an honorable task to be a grass-roots statistician, and many an unreadable doctoral dissertation might be equally unreadable, but a hundred times more useful, if directed to bettering our humble arithmetical knowledge of something. Such research is complicated detective work, often demanding the highest and broadest talents. Secondly, there is much theoretical work to be done on how to use bad data. In some cases, for example, guesswork can actually increase the accuracy of results. The author is much too pessimistic and unimaginative on this. Thirdly, economists in underdeveloped countries must specially resist temptation. None of the latest and most glamorous techniques is worth applying in such countries, simply on Morgensternian grounds. Whatever the cost in self-respect and academic advancement, the intellectual sacrifice must be made.

Lastly, economists must cease to run miles ahead of what they know, developing techniques that cannot be used, and theories that cannot be tested, on actual numbers. It is very unfortunate that certain great minds of the past, prone to this same vice, have been by chance “vindicated” decades or centuries later. Does it really stand to Leonardo’s credit that he drew a flying-machine that obviously wouldn’t fly? Or even to Lobachevsky’s that he invented a geometry with no current applications? Why not honor all the similar lucubrations that turned out not to foreshadow some reality?—After all most of the Great Anticipators merely fluked it. In the name of just such free and pure research mathematical economists waste a powerful amount of their energy and the public’s money. Let them tell us instead how much is smuggled across the Canadian border, or consumed on the farm, or staked on the numbers game.