• © 2008 by American Society of Clinical Oncology

Randomized Trials in Oncology Stopped Early for Benefit

Investigators conducting randomized controlled trials (RCTs) who find a large treatment effect before the planned completion of the study may terminate the study early. A systematic review of RCTs that were stopped early for benefit recently concluded that many of these trials not only failed to adequately report those factors, which informed the investigators’ decision to terminate the study prematurely, but also reported implausibly large treatment effects.¹ Despite their propensity to overestimate the true treatment effect, particularly when few end points had occurred, these trials are becoming more prevalent and are often published in high-impact journals.

The United Kingdom Medical Research Council's (MRC's) 12th acute myeloid leukemia trial (AML12), which sought to determine whether an additional course of consolidation chemotherapy improved survival when compared with four courses of chemotherapy, illustrates the problem of overestimation of treatment effects as a result of stopping early for apparent benefit.² An interim analysis demonstrated a significant survival advantage in favor of five cycles of chemotherapy. As the difference in survival did not meet the stopping rule, the study was continued, and the next interim analysis was performed 6 months later. This interim analysis again found that five cycles of chemotherapy was associated with improved survival (hazard ratio [HR], 0.55; 95% CI, 0.38 to 0.80; P = .002). Given the implausibility of these results (because of the larger than expected effect on mortality and its inconsistency with the reduction in the risk of relapse, the key postulated mechanism for improved survival), the investigators continued the trial despite the apparent difference in survival. After random assignment of more than 1,000 patients, a significant survival advantage for five courses of consolidation chemotherapy could not be demonstrated, with an HR of 1.09 (95% CI, 0.87 to 1.37; P = .4). The apparently significant results observed after few end points (ie, deaths) had occurred may be explained by data analysis at a “random high,” when a disproportionate number of events had occurred in the experimental arm by chance. Had this trial been stopped early and its results used to justify an additional course of consolidation chemotherapy, subsequent patients with acute myeloid leukemia would have been exposed to a costly, potentially toxic and ineffective therapy.

The extent to which medical oncology RCTs that are stopped early for benefit manifest problems of inadequate reporting and likely effect overestimations, remains uncertain. Therefore, we examined in detail the medical oncology RCTs previously reported in the larger systematic review.¹ We reviewed the study characteristics, features related to the decision to monitor and stop the study early (sample size, interim analyses, monitoring and stopping rules), the number of events, and the estimated treatment effects reported in the 29 RCTs in medical oncology that had been stopped early for benefit (Appendix, online only).

Premature termination of RCTs in medical oncology is becoming more prevalent, as 52% of those RCTs stopped early for benefit were published in the years 2000 to 2004. Furthermore, 41% were published in three high-impact medical journals (New England Journal of Medicine, Lancet, and JAMA). Most trials (72%) reported a primary source of funding, 76% of which were primarily funded by a not-for-profit or government organization; 24% were industry-sponsored. RCTs reported (1) the planned sample size (n = 23), (2) the interim analysis at which the study was terminated (n = 18), and (3) whether the decision to stop the study prematurely was informed by a stopping rule (n = 17). Only 13 (45%) reported all three.

The relative risk (RR), either reported or calculated, and total number of events were available for all those RCTs reporting dichotomous end points (n = 27). The median RRs and total number of events were 0.54 (interquartile range, 0.3 to 0.7) and 61 (interquartile range, 23 to 144), respectively. Nine of the 12 RCTs that reported a RR ≥ the median RR reported more than the median number of events. Conversely, the majority of RCTs (73%) that found an RR less than the median also reported fewer than the median number of events, suggesting that RCTs stopped early after only a few events accrued tend to report large treatment affects. Indeed, the odds of reporting a large treatment effect (ie, a RR less than the median) were 12.2 times (95% CI, 2.0 to 75.1) greater when the studies accrued fewer than the median number of events than when they stopped later, after more events had occurred. The correlation between the number of events and the apparent treatment effect, expressed as an RR, was highly significant (r = 0.75; P < .0001). Therefore, the risk of significantly overestimating the treatment effect diminishes when the number of events accrued is large. While preliminary explorations suggest that subsequent trials confirmed the statistical significance of 64% of trials stopped early for benefit in oncology, we are concerned with the overestimation of the true treatment effect and how this could affect risk-benefit tradeoffs necessary for sound clinical decision making.³ Given the strong correlation between the apparent treatment effect and the number of events, it would seem prudent to perform interim analyses only after a sufficient number of events have occurred so as to reduce the likelihood of overestimating the true treatment effect.

A number of factors may motivate the various stakeholders involved in the conduct of randomized trials to terminate a study prematurely as soon as a difference emerges between the experimental and control arms. Clinicians, investigators, patient advocacy groups, and data monitoring committees may hope to offer patients randomly assigned to the control arm the superior treatment; the prospect of a high-impact publication may motivate both investigators and journal editors; fiscal considerations may influence funding agencies. Data monitoring committees are in a particularly difficult position when they face trials indicating possibly large benefits. If the trial continues, many patients will continue to receive a potentially inferior treatment. If the trial stops, clinicians and patients, including participants in the trial, may prematurely adopt a new therapy of questionable efficacy and unclear safety.⁴ Thus, stopping trials early for benefit may be unethical, as a trial that produces an inflated treatment effect loses its scientific and societal value, and risks failing to honor the trial participants and to benefit future patients.⁵

Clinicians should interpret RCTs stopped early for benefit with caution, as the large treatment effect observed may be a chance finding. When a trial's stopping boundaries have been exceeded due to an implausibly large treatment effect, investigators may choose to continue a trial only to find that no important difference between treatment arms exists. The MRC AML12 trial, cited earlier, is one such example. Furthermore, trials that are terminated prematurely and report implausibly large treatment effects may have considerable impact on practice, as illustrated by trials investigating the use of preoperative beta-blockers.^6-8

Some experts have suggested handling the overestimation of the results of a trial stopped early for benefit using statistical adjustments that take into account the number of interim analyses performed. While we are not aware of strong evidence supporting this approach, none of the truncated RCTs in medical oncology we reviewed reported an adjusted estimate of the treatment effect or indicated that stopping early and the resulting bias were significant limitations of the study. The identification of study characteristics that are able to predict the likelihood of overestimating the true treatment effect and determine the degree to which truncated RCTs may overestimate this effect, are areas of ongoing investigation.

Clinicians must interpret RCTs stopped early for benefit with caution, especially when information about the decision to stop early is not provided and few events have occurred. Failure to do so may result in prematurely translating the findings reported in these trials into clinical practice. Our ongoing research in this matter will help identify when and to what extent results from trials stopped early for benefit are in fact too good to be true.