- © 2003 by American Society of Clinical Oncology
Beyond the Development of Health-Related Quality-of-Life (HRQOL) Measures: A Checklist for Evaluating HRQOL Outcomes in Cancer Clinical Trials—Does HRQOL Evaluation in Prostate Cancer Research Inform Clinical Decision Making?
- Fabio Efficace,
- Andrew Bottomley,
- David Osoba,
- Carolyn Gotay,
- Henning Flechtner,
- Sven D’haese and
- Alfredo Zurlo
- From the European Organization for Research and Treatment of Cancer (EORTC), Quality of Life Unit and Genitourinary Unit, EORTC Data Center, Brussels; QOL consulting, West Vancouver, BC, Canada; Cancer Research Center of Hawaii, University of Hawaii, Honolulu, HI; and Clinic for Psychiatry and Psychotherapy of Children and Adolescents, University of Cologne, Cologne, Germany.
- Address reprint requests to Fabio Efficace, MSc, Ave E Mounier 83, 1200 Brussels, Belgium; e-mail: fef{at}eortc.be.
Abstract
Purpose: The aim of this study was to evaluate whether the inclusion of health-related quality of life (HRQOL), as a part of the trial design in a randomized controlled trial (RCT) setting, has supported clinical decision making for the planning of future medical treatments in prostate cancer.
Materials and Methods: A minimum standard checklist for evaluating HRQOL outcomes in cancer clinical trials was devised to assess the quality of the HRQOL reporting and to classify the studies on the grounds of their robustness. It comprises 11 key HRQOL issues grouped into four broader sections: conceptual, measurement, methodology, and interpretation. Relevant studies were identified in a number of databases, including MEDLINE and the Cochrane Controlled Trials Register. Both their HRQOL and traditional clinical reported outcomes were systematically analyzed to evaluate their consistency and their relevance for supporting clinical decision making.
Results: Although 54% of the identified studies did not show any differences in traditional clinical end points between treatment arms and 17% showed a difference in overall survival, 74% of the studies showed some difference in terms of HRQOL outcomes. One third of the RCTs provided a comprehensive picture of the whole treatment including HRQOL outcomes to support their conclusions.
Conclusion: A minimum set of criteria for assessing the reported outcomes in cancer clinical trials is necessary to make informed decisions in clinical practice. Using a checklist developed for this study, it was found that HRQOL is a valuable source of information in RCTs of treatment in metastatic prostate cancer.
THE NUMBER of studies in medicine that incorporate health-related quality-of-life (HRQOL) assessment has continued to grow since 1970.1 At the present time, there is no single definition of HRQOL. Nevertheless, there is a broad consensus that it refers to the physical, psychologic, and social functioning of patients and the impact of disease and treatment on their abilities and daily functioning.2,3
In cancer research, the number of clinical trials incorporating HRQOL as part of their trial design, either as a primary or secondary end point, has been growing over the last decade.4 While the number of cancer clinical trials including HRQOL assessment is increasing, there is also evidence that the potentially invaluable insights that HRQOL data provide into the treatment and care of patients may not be adequately reported.4–,6
Recently, Levine and Ganz7 argued that despite the fact that thousands of cancer patients have had their HRQOL measured by a number of HRQOL instruments in clinical trials, there are few examples of HRQOL outcomes that have influenced individual patient decision making or treatment policies. Hence, bridging the gap between HRQOL research and clinical practice is still an immediate challenge.
At present, a wide range of HRQOL instruments is available for use with cancer patients, ranging from general to disease-specific tools (which are also suitable for specific cancer populations undergoing specific treatments).8 It is reasonable for instruments to vary, depending on the concepts or experiences they intend to capture. As an example, assessing HRQOL in breast cancer patients who are free of disease a year after surgery may be somewhat different from measuring HRQOL in patients with advanced metastatic lung cancer. On one hand, the ample availability of measures has helped researchers to include HRQOL as an outcome in clinical trials, but on the other, the proliferation of such measures often turns out to be a barrier for decision makers having to evaluate HRQOL across disease stages and to compare the impact of alternative therapies.9
In this article, we present a systematic approach to evaluating the methodologic rigor of the HRQOL assessment in randomized controlled trials (RCTs) and apply this to prostate cancer. We examine RCTs of prostate cancer therapies that incorporate HRQOL as a study end point and evaluate the impact of HRQOL findings on treatment decision making in prostate cancer, one of the most common solid tumor malignancies and the second leading cause of cancer death in American men.10 We focus on RCTs because they represent the gold standard for evaluating new medical treatments, for development of decision-making policy, and for planning new treatment approaches.11 Within the RCT setting, HRQOL assessment has the potential to provide a comprehensive picture of the possible risks and benefits of new medical treatments.
MATERIALS AND METHODS
Study Identification
A literature search for studies meeting the criteria was undertaken on the following databases: MEDLINE (1980 to 2001), CancerLit (1980 to 2001), and the Cochrane Controlled Trials Register (Cochrane Library 2000, Issue 3). In an effort to retrieve all relevant studies, significant articles listed as references were also included. The comprehensiveness of the search was enhanced by including additional key words relevant to prostate patients (eg, “sexual functioning” or “sexual interest”), in addition to “quality of life” (or “HRQOL” or “QoL” or “health status” or “patient-reported outcome”) and “prostate cancer.” The search strategy for MEDLINE and CancerLit databases was restricted to “randomized controlled trials.” No restriction in the search-field description or in the language of the article was performed. Conference abstracts were excluded given the lack of comprehensive information regarding the HRQOL trial design and the lack of peer review of all the data. This methodology of selecting RCTs in prostate cancer has been previously reported.12
Criteria for Considering Studies
Types of participants.
Participants were patients diagnosed with prostate cancer regardless of the stage or the grade of the tumor. Studies on patients with benign prostate hyperplasia and patients undergoing screening procedures were not eligible.
Types of intervention.
Interventions included any RCTs comparing medical treatment (eg, radiotherapy, systemic therapy, surgery) regardless of the intervention. Any studies dealing with psychologic intervention or any kind of intervention other than medical treatments were excluded. The rationale of the choice ties in with the scope of our study, namely, understanding whether the inclusion of HRQOL as part of the trial design contributes to treatment decision making.
Types of outcome measures examined.
HRQOL and traditional clinical outcomes were the main outcome measures examined. Specifically, HRQOL has been reported as an important issue in clinical trials with prostate cancer patients.13 Any studies including HRQOL either as a primary or secondary end point were considered. Only HRQOL self-reported patient measures were taken into account, excluding studies exclusively assessing HRQOL by means of physician-assessed measures, such as the Karnofsky performance status index. This rationale is based on evidence showing considerable variability in results between clinician and cancer patient HRQOL reported outcomes.14–,16 Recent studies on prostate cancer patients have shown that physician ratings may not accurately reflect the functional health and symptom experiences of the patient.17 Also, there is a general consensus that patients are the prime source of information on individual HRQOL, making a patient self-reported assessment desirable in clinical trials.18,19
Types of studies.
All RCTs comparing different medical treatment modalities and symptom management enrolling no fewer than 100 prostate cancer patients (combined arms) published from 1980 to 2001 were studied. Trials not reporting both clinical and HRQOL outcomes (either in the same article or in a separate analysis) were excluded.
Methods of Evaluation of the Studies
At the present time, there are no agreed international standards for assessing HRQOL in RCTs.20 However, attempts to create a brief HRQOL checklist to assess the level of reporting of HRQOL have been previously made by Lee and Chi.6 Although their checklist is helpful in identifying the key elements important in developing and reporting HRQOL findings, we found some weaknesses for our current purpose. For example, having a baseline HRQOL as well as an evaluation of the adequacy of HRQOL domains covered by the measure used is crucial to interpreting results in a clinical trial context, but these were not taken into account. Furthermore, some of Lee and Chi’s categories,6 such as “method of QL data collection,” included multiple factors that we believed should be considered separately; some others were not relevant in helping to interpret the robustness of the reported HRQOL outcomes. Hence we devised a minimum standard checklist for evaluating HRQOL outcomes in cancer clinical trials (Table 1⇓) to provide a guide for the evaluation of cancer clinical trials that include HRQOL assessment. We suggest that studies that meet our checklist criteria are more likely to result in robust HRQOL outcomes.
The checklist comprises the basic and essential issues that a given trial should address to have reliable outcomes. On the basis of good practice in reporting HRQOL,6,21–,24 we extracted 11 key issues (with consensus of F.E., A.B., and D.O.) for ranking the studies to have a minimum criterion for reliable HRQOL outcomes in a cancer clinical trial setting. Most of the items were devised to have a dichotomous answer (yes or no), simplifying the evaluation process. Although the items included in the checklist are self-explanatory and a brief description is also provided (Table 1⇓), some issues deserve further clarification. To satisfy the “a priori hypothesis stated” criterion, a predefined HRQOL end point for the study was requisite, unless the study explicitly stated that the purpose of the trial was to explore which HRQOL aspects are affected by treatment. Without such a priori hypothesis, clinical trials might generate spurious HRQOL results because of multiple significance testing and differences between treatment groups can be exaggerated. A positive rating for “psychometric properties reported” required information on the validation process (and related psychometric data) of the HRQOL instrument if the trial used a measure not previously reported in the literature or if it used an amended version of a previously validated measure. If a trial used the original version of a previously validated measure, we rated “yes” as well. To satisfy the “adequacy of domains covered” criterion, the study needed to address at least the HRQOL aspects related to a generic cancer population. In the case of a trial with a predefined HRQOL end point, we assessed as “yes” if the domain related to the specific research question of the trial was covered by the measure used. The trial research question may be unanswered if an HRQOL measure is not sensitive enough to detect HRQOL changes for the specific target population of the study. “Presentation of results in general” was scored as “yes” if any attempt was made to describe and/or discuss the HRQOL results. This issue was evaluated as “no” if HRQOL outcomes were not mentioned at all when discussing the results of the study.
Because we also realized that some of the items of the checklist were likely to have a higher impact than others in determining the reliability of the results, we surveyed an ad-hoc panel of 30 HRQOL researchers (including clinicians, psychologists, and statisticians), with international expertise in oncology, to identify the most critical issues. All panel experts were identified on the grounds of their international expertise in the field and their relevant publications related to quality-of-life issues in cancer research. They were independently asked to rate the 11 items of the checklist from 1 (most important) to 11 (less important). Although general comments of the experts recognized the basic importance of all the issues included in the checklist, their final ratings identified some priority issues, such as “baseline compliance reported,” “psychometric properties reported,” and “missing data documented” (see Appendix).
Because the purpose of this study was to evaluate whether high-quality HRQOL outcomes had an impact on clinical decision making, we needed to operationally define the construct of “high quality.” Our rule of thumb for this article was that HRQOL data would be judged as high quality if at least eight of the 11 criteria were satisfied; furthermore, three of the eight or more criteria satisfied needed to be the three high-priority concerns identified by the experts (“baseline compliance reported,” “psychometric properties reported,” and “missing data documented”). We required these three issues as mandatory. If one or two of the 11 items on the checklist were assessed as not applicable, the cutoff was set at seven or six criteria, respectively (but the three mandatory criteria were still required to be met). Studies that addressed at least eight issues but did not take into account the mandatory items, as well as the ones addressing fewer than eight issues, were judged as having some possible reported methodologic constraints.
Two reviewers (F.E. and A.B.) independently reviewed all identified RCTs. Where disagreement about the possible inclusion of an article occurred, the two reviewers re-examined the study to reconcile any differences.
The trials were examined in terms of the effects of the clinical interventions on both HRQOL and biomedical outcomes. Traditional clinical end points (such as overall survival, progression-free survival, or prostate-specific antigen response to treatment) were only considered when the statistical significance of these outcomes, based on planned analyses, was reported in the article. We did not consider or discuss results obtained in ad-hoc subanalyses.
RESULTS
Two reviewers (F.E. and A.B.), who were neither authors nor coauthors of any of the 24 RCTs identified, independently classified them according to the predefined checklist (Table 1⇓), as described above, and agreed on the final classification (Tables 2⇓ and 3⇓). For RCTs with high-quality HRQOL data, we also examined the level of compliance at baseline (Table 2⇓).
The overall level of reporting is provided in Table 4⇓. The two main drawbacks of the studies were a lack of an a priori hypothesis and a lack of a discussion of the HRQOL outcomes in terms of clinical significance. As the aim of this study was to evaluate the impact of these RCTs on clinical decision making, specific methodologic constraints related to the HRQOL assessment methodology used are not reported, but have been previously addressed.12 For the RCTs providing evidence of high-quality HRQOL design, analysis, and reporting (Table 2⇓), future treatment recommendations were extracted according to the traditional clinical and HRQOL outcomes reported in each article. Given some possible methodologic constraints of HRQOL trial design or analysis in some trials (Table 3⇓), treatment recommendations from these trials are not reported, because they are unlikely to provide high-quality HRQOL information supporting decision-making policy. These studies are listed, nevertheless, because they may be useful in future research.
Overall, three studies dealt with early and locally advanced disease,36,42,43 13 studies dealt with locally advanced and metastatic hormone-sensitive patients,27,28,34,35,37,38,40,41,44–47,49–,52 and eight studies dealt with locally advanced and metastatic hormone-resistant patients.25,26,29–33,39,48 Five studies used HRQOL as a primary end point,25,29,30,44,48 whereas the remaining RCTs included HRQOL as a secondary end point.
Treatment Recommendations Based on Methodologically Sound HRQOL Reports
One third of the studies (eight) were judged to possess robust HRQOL trial design features, thus providing confidence in the quality of the findings (Table 2⇓). Two RCTs dealt with hormone-sensitive cancer,27,34 whereas the remaining six studies compared new medical treatments for hormone-resistant cancer. Four studies did not show any differences between treatment arms for the traditional clinical end points,25,27,31,32 and only one reported a borderline difference in overall survival.34 On the other hand, six trials reported evidence of HRQOL differences between treatment arms, thus giving insight for treatment decision making and future research.25,28,29,31,33,34 However, none of these trials used the same measure, thus hampering a comparison of the outcomes and a clear understanding of whether a specific domain can be influenced by treatments. The two studies dealing with hormone-sensitive patients did not provide evidence of traditional clinical benefit for adding either flutamide27 or mitomycin C34 to orchiectomy. On the other hand, they both provided evidence of better HRQOL for orchiectomy-alone treated groups, and one34 also provided evidence of better overall survival.
Tannock et al25 and Osoba et al,26 comparing chemotherapy with prednisone with or without mitoxantrone in symptomatic patients with hormone-resistant cancer, provided evidence of better and longer-lasting HRQOL outcomes for patients treated with mitoxantrone plus prednisone, showing that despite failure to achieve prolongation of survival, HRQOL can be improved. Hence, they recommended treatment with mitoxantrone plus prednisone. Later, Kantoff et al33 also showed some HRQOL and clinical differences, including a longer time to treatment failure, favoring the mitoxantrone plus hydrocortisone arm. Fossa et al,31 comparing flutamide versus prednisone, did not show any survival benefit but provided evidence of better HRQOL outcomes favoring the prednisone group for symptom problems (eg, nausea and vomiting and fatigue), pain, and global health status. For this reason, the authors advocated the use of prednisone as a first-line treatment in patients with hormone-resistant cancer and also recommended exploring the combination with tolerable cytotoxic treatment, as previously done by Tannock et al.25 Small et al29 demonstrated a longer time to disease progression in the group of patients treated with hydrocortisone and suramin, as well as a significantly longer-lasting pain response compared with hydrocortisone alone. The remaining two trials enrolling patients with hormone-resistant cancer, although shown to have a robust HRQOL trial design, failed to demonstrate any significant difference between treatment arms.30,32
DISCUSSION
Using the minimum standard checklist for evaluating HRQOL outcomes in cancer clinical trials, we classified studies according to their methodologic rigor. One third of the studies provided robust information coming from HRQOL analysis, thus supporting informed decision making and planning future RCTs (Table 2⇓). We found that more recent studies tended to be better designed and executed than earlier studies; it is noteworthy that all the higher-quality HRQOL analyses were published between 1998 and 2001 (Table 2⇓), whereas seven of the 16 studies with methodologic flaws (Table 3⇓) were published before this time. Hence it seems that in the last few years, there has been an increasing awareness by researchers of some key HRQOL issues that should be included in outcomes reports.
Overall, none of the studies dealing with early and localized prostate cancer had HRQOL as a primary end point, whereas 50% of the studies dealing with metastatic hormone-resistant patients had HRQOL as a primary end point. Given the lack of significant improvement for survival that can be achieved in this latter group of patients, it is not surprising to note this trend. Only 17% of the trials showed a significant difference in overall survival, and 54% did not show any difference for the minor traditional clinical end points also (eg, prostate-specific antigen response and time to progression). On the other hand, 74% of the studies (testing for HRQOL difference) showed some evidence of better HRQOL favoring one arm of the trial. In some cases, although the trials showed HRQOL differences, the outcomes were compromised because of possible methodologic design constraints (Table 3⇓). Because HRQOL studies often use a large number of variables assessed repeatedly over time and between groups, the possibility of obtaining significant results by chance is likely, even when adjusting for multiple comparisons. A key consideration for future studies is to select a limited number of HRQOL indicators before commencing the trial, possibly basing this selection on previous related trials or on specific a priori research hypotheses.
Eight (one third) of the studies provided high-quality evidence from the HRQOL analysis and supported informed decision making for planning future RCTs (Table 2⇓). Six of these eight robust RCTs26,27,29,31,33,34 demonstrated significant HRQOL differences between treatment arms, thus providing insight for treatment planning. The added value that the HRQOL information provided was important, because most of these studies provided quite limited evidence for the differences in the traditional clinical end points, and only one reported a borderline difference in overall survival.34 The remaining two studies did not show differences between groups.30,32 Interestingly, in those studies where differences in HRQOL were tested, most provided valuable information in terms of understanding the course and nature of the disease and treatment. Thus the benefit of including HRQOL assessments might give a comprehensive picture of the whole treatment benefit, as well as the perspective of the patient himself.
The numbers of cancer clinical trials incorporating HRQOL measurement have increased substantially in recent years. Although the number of published studies evaluating HRQOL is constantly increasing, few tools are available to help clinicians and readers in general in interpreting HRQOL outcomes and distinguishing those that are likely to stand the test of time from those that are not. However, there is also compelling evidence of the impact of HRQOL assessment on clinical practice.7
At present, there is a lack of a gold standard against which to rate the quality of the HRQOL evaluation in RCTs.20 Although the Consolidated Standards of Reporting Trials guidelines11 are useful to evaluate the overall quality of reports of RCTs, they do not specifically address HRQOL issues; hence they are unlikely to provide a standard against which the quality of the reported HRQOL outcomes may be compared. Therefore, we devised the minimum standard checklist for evaluating HRQOL outcomes in cancer clinical trials, basing the selection criteria on good practice for reporting HRQOL.6,21–24,53The adoption of a brief checklist such as the one proposed in this article could also help raise the standard of HRQOL reporting in cancer clinical trials and could facilitate more systematic evaluation of the clinical impact of HRQOL assessment in cancer research.
The checklist presented in this article has a number of strengths. It provides easy minimal criteria by which to evaluate the key issues that a study should address if it is to be considered reliable for the HRQOL outcomes, and it should be helpful to clinicians (and readers in general) when interpreting results from a given cancer clinical trial that includes an HRQOL evaluation. However, the checklist has limitations also. It is important to note that this checklist is not meant to be a comprehensive guideline for the optimal design of HRQOL assessment in clinical trial protocols, as this issue has been previously addressed.21,22,53 It is only applicable to studies evaluating patient-reported outcomes, as it is well recognized that this is the most appropriate method for assessing HRQOL.14–19,54 Although the minimum standard checklist for evaluating HRQOL outcomes in cancer clinical trials consists of 11 items, we feel that a minimum coverage of eight of the 11 items is required to judge a study as having high-quality outcomes. We feel this proportion is a reasonable rule of thumb, given the fact that we only addressed the basic and essential key issues. However, the cutoff of eight should not be seen as fixed, nor should the number of issues be considered as mandatory. The number of criteria that are critical in a given study might vary, and the cutoff could be tightened or relaxed accordingly. In fact, in the current analysis, we also examined the impact of tightening or relaxing the cutoff by one item for the determining proportions of qualifying studies (Table 2⇓). With a cutoff of nine or seven, we could have included either six (25%) or nine (37%) studies, respectively, in contrast to the eight (33%) in the current analysis.
We also realize that HRQOL design also greatly depends on the context and the specific research question of the trial; hence good reports may have different emphases and some issues might have different relevance according to the specific study questions. For example, “timing of assessment documented” might be important in trials assessing different chemotherapy regimens, whereas “rationale for instrument” might be important in trials not using an appropriate measure for the target population of the trial. On these grounds, it might be difficult to set some predetermined mandatory issues, and it might be reasonable to decide case by case if certain items of the checklist should be considered as mandatory. In any case, our ad-hoc survey of experienced HRQOL researchers suggested that some of the items on the checklist will have less of an effect, whereas others may have a higher impact on the reliability of the HRQOL outcomes (see Appendix). A limitation of this article is that we arbitrarily decided to have three mandatory issues and an arbitrary cutoff to determine high-quality HRQOL methodology for all studies; thus some additional relevant information might have been overlooked. On these grounds, we recognize that the checklist might benefit from further validation, and we invite other researchers to contribute to this effort. Nevertheless, the checklist can be considered as a minimum standard, and its future value will lie in its pragmatic use in helping clinicians to easily interpret HRQOL outcomes from a given clinical trial. Also, journal policies could favorably influence the quality of HRQOL research by requiring authors submitting HRQOL papers to meet minimum standards or provide an explanation of why these issues were not addressed. In this respect, the present checklist, as well as other previously published ones, might be useful for raising the quality of HRQOL reporting in a clinical trial context.
If RCTs that include HRQOL evaluation are to fulfill their potential of allowing health care providers to make informed inferences about the overall validity of the treatment, researchers should continue to improve the design of HRQOL studies and the reporting of key HRQOL issues. The present checklist has the potential of significantly reducing bias in the interpretation of the outcomes in those cancer clinical trials that report HRQOL assessment and helping raise the standards of reporting. This is a crucial aspect in HRQOL research, because one of the main challenges is to bridge the gap between research and clinical practice, and this is likely to be achieved by robust outcomes from clinical trials.
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST
The authors indicated no potential conflicts of interest.
APPENDIX
Results of Survey of Panel Experts*
Minimum Standard Checklist for Evaluating HRQOL Outcomes in Cancer Clinical Trials
Randomized Controlled Trials With Robust HRQOL Design
Randomized Controlled Trials With Possible HRQOL Design Constraints
Level of Reporting According to the Minimum Standard Checklist for Evaluating HRQOL Outcomes in Cancer Clinical Trials
Acknowledgments
We thank Sheila Sanderson Scott for the review of the manuscript and the panel of experts in quality-of-life research for their willingness in reviewing the checklist. A list of the panel experts is available from the authors.
Footnotes
-
F.E. is supported by the Camilla Samuel Fellowship in memory of Lady Grierson.
-
The contents of this article are solely the views of the authors and do not necessarily represent the official views of the respective organizations.
-
Presented at the Thirty-Ninth Annual Meeting of the American Society of Clinical Oncology, Chicago, IL, May 31–June 3, 2003.
- Received December 20, 2002.
- Accepted July 1, 2003.
