Page 35 - JCTR-9-4

Shuster | Journal of Clinical and Translational Research 2023; 9(4): 246-252 251
Shuster [1] reported on a small sample of 32 highly cited past is a very common occurrence. Note that if Assumption A4 is false,
meta-analyses that used mainstream methods for relative risk and every weighted combination is estimating a different overall target
found major disparities in eight (25%). It is fortunate that this is population mean, and the mainstream analysis of the sequence
not higher, but this is not good enough for trust in mainstream will push the overall estimate toward targeting the unweighted
methods. mean in the urn. The key question when looking at the difference
An analysis of the 31 of these studies reported in Shuster et al. [6] between the mainstream point estimate and the unweighted point
and Shuster and Walker [7] (the 32 study’s reanalysis had a few estimate is whether it is simply sampling error or is it bias induced
nd
studies added but trended as the 31 we analyzed) also dispel the one by failure of Assumption A5 (that the presumption that weights
remaining scientific as opposed to traditional reasons for using the and effect sizes are uncorrelated is false). There is no way to be
mainstream: The mainstream might produce on average narrower sufficiently certain, as there is no adequately powered statistical
confidence intervals. If you analyze the natural logs of the ratios of test that can prove the lack of such a correlation.
the lengths of the confidence intervals (the traditional way to analyze Note that the phenomenon of seeing two sets of data with the
non-negative ratios) and treat the studies as a random sample of same signals but noise level of the second reduced by a common
highly cited meta-analyses, we obtain an estimate of the population factor and turning the result from significant to not significant
ratio of widths (Mainstream: Shuster) of 1.10 (Mainstream wider) cannot occur in the common statistical methods: t-tests, analysis of
with 95% confidence interval from 0.93 to 1.29. The mainstream variance or covariance, regression, logistic regression, frequency
plausible mean in the total population of such potential reanalysis tables, or Cox regression (survival analysis).
ranges from slightly shorter to substantially longer. 11.4. Recommendations for meta-analysis of clinical trials with
Further, due to mainstream proponents’ concerns about its tabular data
normal approximation, newer versions of CMA have added a
t-option (degrees of freedom number of studies less one) that (1) Use random effects rather than fixed effects; (2) With fewer
can be used instead of the normal approximation. When we than five studies, do a Systematic Review, not a formal meta-analysis,
replaced the normal with the t, the new mainstream methods were since the large sample distribution of the estimates should not be
significantly less accurate than the survey-based method. For trusted; (3) with 5-20 studies, issue a limitation that the number of
our sample of 31 meta-analyses, in the log scale, the mainstream studies is small with a caveat on successful vetting for relative risk;
averaged 30% wider than the survey-based methods, with 95% (4) use Shuster [1] until new methods become available; (5) if you
confidence interval from 9% wider to 54% wider. This is yet have individual patient data, note that the off-label implications for
another strong reason not to use the mainstream. the mainstream for tabular data may or may not apply to individual
Note that diagnostic test information, such as Cochran’s Q, patient data. Shuster [1] can still work if any of Assumptions A2,
I , and Egger’s test for selection bias, as described in Borenstein A4, and A5 are used in the individual patient analysis, potentially
2
et al. [8] is not relevant to the validity of Shuster [1]. endangering its evidence base; (6) a biostatistics group with access
Shuster et al. [6], with a substantial number of simulations, to supercomputers should conduct large simulations along the lines
found that when the target population relative risks in the two of Shuster et al. [6] for the robustness of the T-approximation of the
universes were the same (ratio and equal weighting), the ratio survey sampling method for differences in means or proportions; (7)
method had an average confidence interval length reduction of a parallel width of confidence interval comparison on mainstream
about 10% compared to equal weighting. versus survey sampling should be done, and (8) if you have
published a meta-analysis that had a substantive influence on public
11.3. The mainstream’s moving target health policy, consider conducting an equal weighted analysis on

Suppose we have a sequence of meta-analyses where the the log of the relative risks or differences in means or proportions to
urns described for obtaining the true study-specific effect sizes see if this new analysis supports your original conclusions. If they
(Assumption A1) are identical, but each member of the sequence do not, consider writing a report, using the recommended methods
has within-study variances of 90% of the previous member of of Shuster [1].
the sequence. Under the mainstream model, all of these meta- 12. Conclusion
analyses have the same true effect sizes, namely the unweighted
mean effect size in the urn. The true mainstream variance of the Based on a reasonable fear gleaned from examples 1 and 2, the
global effect size estimate is the sum of its between-study variance continued use of the mainstream methods is threatening to public
(Same for all members in the urn) and the within-study variance health interests. In example 1, despite the published mainstream
(which will shrink toward zero as you get later into the sequence). inference, there is no evidence that a widely used invasive
Thus, the mainstream estimates will become closer and closer to intervention is effective. Example 2 had the survey sampling
the unweighted estimator as we get further into the sequence. You method been available and utilized, and the use of rosiglitazone
therefore cannot rule out an artifact of where you might be in the in Type II diabetes would likely have been eliminated far earlier
sequence for any meta-analysis where the qualitative conclusions than what occurred, saving a substantial number of cardiac events
of the mainstream differ from the unweighted (one significant and from happening. Other similar misjudgments stemming from the
one not). The Neto et al. [4] example is one case of this, but this mainstream methods are all but certain to occur in the future.
DOI: http://dx.doi.org/10.18053/jctres.09.202304.22-00019