It is the mark of an educated man to look for precision in each class of things just so far as the nature of the subject admits.
Aristotle
UK gastroenterologist and fellow contrarian James Penston has published two books critical of mainstream clinical research methodology. The most recent one, called Stats.con, deals with the misuse of statistics in medicine and expands on the topic of his first book published in 2003 under the title Fiction and Fantasy in Medical Research: The Large-Scale Randomised Trial.
Penston’s two books provide a compilation of the methodological shortcomings, fallacies, and abuses of clinical research methods that rely heavily or exclusively on statistical analysis. Many of these errors have already been identified in academic journals, but Penston condenses this information and gives excellent references. Fiction and Fantasy, for example, dedicates a chapter to the overwhelming difficulties of randomization and reviews the pertinent evidence that many—if not most—clinical trials fail in the criterion of internal validity. Another chapter details the difficulties of deriving generalizable knowledge from large-scale clinical trials. Stats.con also offers a good review of the “frequentist vs. Bayesian” debate as it applies to clinical research, and discusses the shortcomings of each approach. Both books also devote chapters to review the most egregious and well-documented cases of fraudulent research and identify the many ways in which abuse can hide behind the screen of statistical manipulation.
But Penston’s main line of criticism against statistical medicine is the seemingly paradoxical idea that large scale clinical research paradigms are contrary to traditional scientific formalism. Penston spends many pages in both volumes to review the historical development of the so-called “scientific method,” placing particular emphasis on David Hume’s philosophy. Penston argues that claiming to determine causality by identifying small but statistically “significant” differences induced by a given intervention is a far cry from the tenets of the theory of regularity and causality proposed by Hume.
There are two problems with this thesis. While Penston is correct that large-scale clinical research methods practically (and probably inherently) lack the rigor commonly associated with scientific experimentation, his opinion that medicine’s great advances of the past are primarily due to the application of the scientific method is questionable. Setting aside the theoretical debate regarding the substance and determinants of scientific inquiry (and Hume’s ideas), much in medicine has been discovered by individuals applying their sense of observation and reasoning in informal if not completely disorganized ways. Human biology, pathology, and behavior, complex and unpredictable as they are, necessarily evade study under strict scientific rigor. (And conversely, statistics-based methods have been used very successfully in a number of settings such as experimental biology or agriculture, contradicting Penston’s argument that seems to be directed against any statistical-based research.)
Large scale clinical studies backed by statistical analysis were precisely developed to get around the difficulties of controlling individual human characteristics so as to facilitate the yield of objective data. But in so doing, these methods necessarily moved the locus of investigation from the individual to the population or group. They fail, not for being unscientific, but for applying scientific methods to a subject rather impervious to this line of investigation. And it is primarily by obfuscating the dichotomy between group knowledge and individual knowledge that proponents of statistics-based research greatly and inexcusably exaggerate the value of the information obtained.
Nevertheless, Penston’s books are remarkable achievements for a lone dissenter in clinical practice. Each chapter is well organized and documented with all the pertinent references. The introductory pages on statistics address the interesting etymology of the word and its relationship to “matters of the state.” And readers of the books will not be put off by any technical jargon or conceptual difficulties. The subject is easily presented and accessible even for those with phobias for numbers. Notwithstanding the comments above, the discussion on scientific method is stimulating, particularly for readers like me who are not particularly well versed in the history of epistemology. All and all, two great books for those who seeking good arguments in the intellectual fight against Leviathan.
