Page 30 - JCTR-9-4

Journal of Clinical and Translational Research 2023; 9(4): 246-252

Journal of Clinical and Translational Research

Journal homepage: http://www.jctres.com/en/home

ORIGINAL ARTICLE

Meta-analysis of clinical trials in the 2020s and beyond: a paradigm shift

needed

Jonathan J. Shuster*
Department of Health Outcomes and Bioinformatics, College of Medicine, University of Florida, Gainesville, Florida 32605, United States of America

ARTICLE INFO ABSTRACT

Article history: Background: A peer-reviewed meta-analysis methods article mathematically proved that mainstream
Received: February 15, 2022 random-effects methods, “weights inversely proportional to the estimated variance,” are flawed and
Revised: April 11, 2023 can lead to faulty public health recommendations. Because the arguments causing this off-label
Accepted: June 10, 2023 (unproven) use of mainstream practices were subtle, changing these practices will require much clearer
Published online: July 12, 2023 explanations that can be grasped by clinical and translational scientists. There are five assumptions
underlying the mainstream’s derivation of its statistical properties. This paper will demonstrate that
Keywords: if the first is true, it follows that the last two are false. Ratio estimation, borrowed from classical
clinical trial survey sampling, provides a rigorous alternative. Papers reporting results rarely fully disclose these
meta-analysis assumptions. This is analogous to watching TV ads with the sound muted. You see high quality of life
random effects and do not hear about the complications. This article is a poster child for translational science, as it
takes a theoretical discovery from the biostatistical world, translates it into language clinical scientists
*Corresponding authors: can understand, and thereby can change their research practice.
Jonathan J. Shuster Aim: This article is aimed at future applications of meta-analysis of complete collections of
Department of Health Outcomes and randomized clinical trials. It leaves it to past authors as to whether to reanalyze their data. No blame
Bioinformatics, College of Medicine, for past use is assessed.
University of Florida, 2026 NW 34 Ter, Methods: By treating the individual completed studies in the meta-analysis as a random sample from
Gainesville, FL 32605, United States of a conceptual universe of completed studies, we use ratio estimation to obtain estimates of relative risk
America. (ratio of failure rates treatment: control) and mean differences, projecting our sample value to estimate
Tel: +1(352)682-0893 the universe’s value.
Email: shusterj@ufl.edu
Results: Two examples demonstrate that the mainstream methods likely adversely impacted major
treatment options. A third example shows that the key mainstream presumption of independence
© 2023 Author(s). This is an Open-Access
article distributed under the terms of the between the study weights and study estimates cannot be supported.
Creative Commons Attribution-Noncommercial Conclusion: There is no rationale for ever using the mainstream for meta-analysis of randomized
License, permitting all non-commercial use, clinical trials.
distribution, and reproduction in any medium, Relevance for Patients: Future meta-analysis of clinical trials should never employ mainstream
provided the original work is properly cited. methods. Doing so could lead to potentially harmful public health policy recommendations. Clinical
researchers need to play a primary role to assure good research practices in meta-analysis.

1. Introduction

As hard as this is to believe, the recent paper, Shuster [1], mathematically proved beyond
any doubt that despite being in common use for over four decades, the mainstream methods
of conducting random effects meta-analyses (true individual study-by-study effect sizes
can differ) are unsound and are likely to produce misleading results that could be threats
to public health. It is commendable if you, as a clinical investigator or reader, would be
skeptical of this statement. Needless to say, because Shuster [1] was controversial, it was

DOI: http://dx.doi.org/10.18053/jctres.09.202304.22-00019