------------ EH.NET BOOK REVIEW --------------
Published by EH.NET (June 2008)
Stephen T. Ziliak and Deirdre N. McCloskey, _The Cult of Statistical
Significance: How the Standard Error Costs Us Jobs, Justice, and
Lives_. Ann Arbor, MI: University of Michigan Press, 2008. xxiii +
287 pp. $25 (paperback), ISBN: 978-0-472-05007-9.
Reviewed for EH.NET by Philip R.P. Coelho, Department of Economics,
Ball State University.
Ziliak and McCloskey have written a fine book of 24 chapters, a
reader's guide, and preface. They write for an "implied" audience of
"keeper[s] of numerical things" to persuade them that: "Statistical
significance is not the same thing as scientific finding. R-squared,
t-statistic, p-value, f-test, and all the more sophisticated versions
of them ... are misleading at best" (p. xv). The authors have
accomplished this and more in a well-researched, written and
documented book. The authors start with a Contents section that
contains a brief pr?cis of each chapter's contents. The pr?cis is an
imaginative and highly useful resurrection of nineteenth- and early
twentieth-century practice; a reader-friendly technique that can be
usefully employed today.
An examination of the Contents is revealing. Directly opposite the
beginning of the Contents section is a photograph of William Sealy
Gosset, the "Student" of the "Student t [commonly truncated to the t]
distribution." In conjunction with an exposition of why statistical
significance is very different from importance or scientific
(economic/historic or whatever) significance, they have written a
paean and brief biography of Gosset. I am convinced that Gosset was
a noble and modest man, a great statistician and intellect who was
shabbily treated by his supposed friend and colleague, R.A. Fisher.
He has also been neglected by historians of science and statistics;
Gosset deserves to be memorialized, and certainly warrants
biographies. That being said, combining two fine books on disparate
subjects (statistical methodology and historical biography) does not
make an even better book.
Ziliak and McCloskey emphatically make their argument against the use
of statistical significance as a proxy for importance in Chapters 1
through 5. The basic difficulty with statistical significance is
that it has been permeated with the mathematical ethos of certainty.
A mathematical "proof" implies a truth (G?del's Theorem is
conventionally ignored) that is invulnerable to time, space, and
reality; it is an abstraction that cannot be falsified using
mathematical epistemology. Relevance, economic importance, and any
metrics other than mathematics are beside the point.
Scientific assertions should be confronted quantitatively with
the world as it is or else the assertion is a philosophical or
mathematical one, meritorious no doubt in its own terms but not
scientific. ...
The problem we are highlighting is that the so-called test of
statistical significance does not in fact answer a quantitative,
scientific question. Statistical significance is not a
_scientific_ test. It is a philosophical, qualitative test. It
does not ask how much. It asks "whether." Existence, the
question of whether, is interesting. But it is not scientific
(pp. 4-5).
In the absence of some measure of how big an effect is, the existence
of an effect reveals nothing of importance about the world of
observational reality.
Ziliak and McCloskey highlight the danger and corruption that flow
from the overwhelming importance placed upon statistical significance
(a measure of existence or lack thereof) by using the tragic example
of Vioxx. Vioxx was a formulation developed by Merck designed to
combat pain. In clinical trials Vioxx had about five times the
number of fatalities as a generic version of a control drug
(naproxen). Because the number of observations did not reach the
appropriate size, the 5 to 1 ratio of excess fatalities caused by
Vioxx was deemed statistically insignificant. (Merck may have
reduced the actual number of fatalities by manipulating the data [p.
29].) Merck's ethics and the clinical/scientific studies of Vioxx
that were sponsored by Merck have been sharply criticized by the
scientific and journalistic establishments. (See the _Wall Street
Journal_, April 16, 2008, p. B4) By simply discarding some
fatalities (on dubious grounds) the 5 to 1 disadvantage in mortality
became statistically insignificant in the submitted trials, and Vioxx
was marketed. It was literally a fatal error that cost Merck
billions of dollars and caused a number of needless deaths.
In the absence of any measure for costs or benefits the standard use
of an acceptance/rejection rate arbitrarily set at five percent is
mindless and/or non-scientific. Five percent of a very large number
(say the world's human population or the GDP of the United States) is
still a large number; and conversely one hundred percent of a
minuscule number is still minuscule. These are not Nobel Prize
winning observations; regardless they are ignored by researchers in a
depressingly large number of disciplines. Ziliak and McCloskey
document (Chapters 5 through 16) the standard statistical conventions
that predominate in publications in a number of journals and
disciplines. The results do not inspire confidence in the scientific
competence of the editors and practitioners. Typically overweening
emphasis is placed on the existence of an effect (statistical
significance) while the magnitude of the effect is either barely
noticed or entirely ignored.
I found other parts of the book fascinating; some are apposite to
their goal of reforming statistical practice (what should be done,
strategies for change), others are not directly germane to their
professed goal (digressions on the life and career of Gosset, Fisher,
Edgeworth, and twentieth century academic politics). The difficulty
with including these digressions is that it makes assigning this book
as ancillary reading for students problematic. What other faults did
I find with the book? 1) Rather than digressions I would like to
have seen a greater emphasis on the analysis of examples, perhaps a
step-by-step numerical approach highlighting the various issues
inherent in statistical "acceptance/rejection." (Vioxx would be a
good case study; another would be the case of black-teenage
unemployment which is statistically "insignificant" yet about 40
percent of the population at risk.) 2) I also found some
deficiencies in the writing; it is too informal and breezy. My
unhappiness with its literary style is strange because McCloskey is
one of the better writers in all of academia today. Regardless, there
are journalistic conventions (I expect done for emphasis) that should
be eliminated; sentences without a noun or a verb are particularly
irritating. Another infelicity is the constant usage of the word
"oomph" instead of importance (relevance, interest, practical
significance, etc.; in the synonym finder I consulted there were over
80 synonyms for the word "significance"). "Oomph" is singularly
distasteful. Perhaps this is a taste unique to me, but I expect that
in five years "oomph" will appear as grating to readers as "groovy"
does now. This book warrants language and style that are more
timeless and less ephemeral. 3) Finally the absence of an index is
the bane of all reviewers. The index may be missing because my copy
is an "advance reading copy," and an index will be in the final
version. If this is not the case, then subsequent printings should
include one.
These are quibbles; this is an important work that deals with a major
problem of statistical analysis in the social, medical and physical
sciences. If you are not aware of the problem, you should be. If
you are aware of the problem, this book is a good compendium of the
problem, real-world issues, and the historical milieu in which the
cult of significance evolved.
Philip R.P. Coelho is a professor of economics at Ball State
University. He has written on and is continuing his study of long-run
economic growth and the impact of parasitic diseases and biology upon
economic growth, history and development. His papers have been
published in the _Journal of Economic History_, the _American
Economic Review_, _Explorations in Economic History_, _Economic
Inquiry_, _Southern Economic Journal_, and other outlets.
Copyright (c) 2008 by EH.Net. All rights reserved. This work may be
copied for non-profit educational uses if proper credit is given to
the author and the list. For other permission, please contact the
EH.Net Administrator ([log in to unmask]; Telephone: 513-529-2229).
Published by EH.Net (June 2008). All EH.Net reviews are archived at
http://www.eh.net/BookReview.
-------------- FOOTER TO EH.NET BOOK REVIEW --------------
EH.Net-Review mailing list
[log in to unmask]
http://eh.net/mailman/listinfo/eh.net-review
|
