INTRODUCTION

Variations in the quality of cancer care are well documented.^1-3 Concern about the national quality of cancer care prompted the Institute of Medicine to review these issues and publish the 1999 report, Ensuring Quality Cancer Care, which called for a national quality monitoring and reporting system.⁴ A key element of quality monitoring is standardized measures of care. These may be used for self-assessment or external review of the quality of care. Payors and purchasers of health care are likely users of standardized measures as they seek data to ensure their beneficiaries are receiving high quality care. Recently, Congress mandated that the Medicare program tie payment to reporting on quality measures (eg, physician-level reporting in the Physician Quality Reporting Program) or the level of performance achieved on quality measures (eg, hospital-level reporting in the Hospital Compare program).⁵ However, payors may not be in the best position to create valid metrics of quality. Professional organizations and provider groups have begun to create metrics based on evidence from research literature and guidelines.

This article reports the methods and results of a program conducted in 2006 to identify quality measures for breast and colorectal cancer by a collaborative group empanelled by the American Society of Clinical Oncology (ASCO) and the National Comprehensive Cancer Network (NCCN). Another major program to identify quality measures was conducted by the National Quality Forum (NQF), a multistakeholder organization that includes payers, providers, consumers, and researchers. The NQF Quality of Cancer Care Measures Project, funded in 2002 and initiated in 2004, aimed to identify valid measures for diagnosis and treatment of breast and colorectal cancer, and symptom management and end-of-life care that could be adopted by the Centers for Medicare and Medicaid Services (CMS) and where gaps could be found, recommend a research agenda.⁶

The ASCO/NCCN project began in 2005 with the organizations agreeing to use their respective data and expertise to establish quality measures for breast and colorectal cancer that capture important processes of care that could be used in pay-for-reporting and other systems, and for which widespread implementation would be feasible. Resources available for the project came from two sources. The first was the 61 quality measures developed during 2001 to 2005 for breast and colorectal cancer defined in a project examining cancer treatment in five metropolitan areas in the United States in the National Initiative for Cancer Care Quality (NICCQ), a collaboration of ASCO, the RAND Corporation, and Harvard University.^7,8 The second was measures for breast cancer based on guidelines developed over the last 10 years by the NCCN⁹ refined for internal quality assessment of care at NCCN member institutions.

This article reports in detail the methods used by the ASCO/NCCN development group and the resulting quality measures. At the conclusion of this project, the NQF process was itself nearing completion. As reported in Appendix 1 (online only), the ASCO/NCCN and NQF groups recognized that they were proposing similar measures and agreed to harmonize these so as to put forth a single set of approved measures to the public. These harmonized measures were approved by the NQF in April 2007.

METHODS

RESULTS

Based on data from the NICCQ and NCCN measures, and using the methodology described earlier, the ASCO/NCCN panels recommended three measures for breast cancer and four measures for colorectal cancer. Table 1^7,14-18 summarizes the quality measures as initially specified by ASCO/NCCN. Table 1 specifies the numerator, denominator, and estimated yearly number of cases in the United States for each measure. In addition, because of concern that an early cancer recurrence occurring during the measurement interval (eg, 1 year after diagnosis for administration of radiation with breast cancer) could bias measurement, relevant literature was used to estimate the small chance of such an early recurrence. Finally, Table 1 presents the variation observed between the five metropolitan regions studied in the NICCQ project for these measures.

Detailed specifications for each measure as finally published after harmonization of the proposed ASCO/NCCN measures with similar measures proposed by American College of Surgeons’ (ACoS) Commission on Cancer (CoC), and endorsed by the NQF, are presented in Appendix Table A1 in Appendix 1 (online only), listing for each measure the population with a clinical indication for the intervention (the denominator), the expected intervention (the numerator), as well as appropriate exclusions and explanatory notes.

The first breast cancer measure examines whether appropriate adjuvant hormone therapy is administered to women with estrogen receptor–and/or progesterone receptor–positive breast cancer. As presented in Table 1, adherence with the similar measure at the sites studied by NICCQ ranged from 85% to 96% with significant differences between the communities where this was tested. A 1-year time window for the initiation of therapy was selected because hormone therapy generally follows completion of surgery, chemotherapy, and radiation. Although not all patients receive all modalities, it was not felt to be feasible to tailor the time window based on treatments received.

The second breast cancer measure determines whether radiation therapy is administered within 1 year after breast conserving surgery (BCS) in women younger than age 70. In NICCQ, adherence to a similar measure was 96% to 99%.⁸ More recent data from NCCN institutions show the rate of radiation after BCS ranged from 89% to 98%.¹⁹ Despite a high level of adherence in pilot testing, this measure was accepted by the panel because improvement of even a few percentage points in this measure among the 130,000 women who undergo BCS in the United States annually would translate into a substantial impact at the population level. The time interval of 1 year for the administration of adjuvant radiation was selected because surgical complications and/or prolonged chemotherapy treatment may result in substantial, but clinically appropriate, delays in the initiation of radiotherapy. Cancer registry data differentiate adjuvant radiation from radiotherapy for recurrent disease, so this measure will be unaffected by treatment administered after recurrence.

The final measure for breast cancer assesses whether adjuvant chemotherapy is administered to women younger than age 70 with stage II or III, estrogen and progesterone receptor–negative disease, a setting with the clearest indication for chemotherapy.¹⁴ The most similar measure evaluated in NICCQ was the initiation of chemotherapy within 8 weeks of surgery among women younger than age 50 with stage II and III cancer and any hormone receptor status. Adherence with the similar NICCQ measure varied widely, from 60% to 91%, across sites.⁸ The ASCO/NCCN panels limited the measure denominator to women with hormone receptor–negative tumors because of recent controversy regarding the utility of chemotherapy in women with hormone receptor–positive tumors.²⁰ The denominator population also is limited to women younger than age 70 because there is insufficient high-quality evidence to support definitive adjuvant chemotherapy recommendations in older women, especially those with comorbid conditions. The measure requires treatment to begin within 120 days of diagnosis. This time frame allows for completion of surgery and appropriate consultation regarding adjuvant treatment options. The panels chose not to specify a list of acceptable drug regimens, or to define a minimum or maximum acceptable duration of therapy, largely because tumor registries only capture information on whether chemotherapy consisted of a single or multiple agents, and they collect the start but not the end dates of chemotherapy.

The panels approved four measures for colon and rectal cancer. The first measure assesses whether patients younger than age 80 with T4N0M0 or stage III rectal cancer begin postoperative adjuvant chemotherapy within 9 months of diagnosis. The panels chose to limit this measure to patients younger than 80 because there is insufficient evidence to support a definitive recommendation for older patients.^21-22 The most similar NICCQ measure assessed whether patients received neoadjuvant and/or adjuvant chemotherapy and found 72% to 92% adherence across the five metropolitan areas studied. The 9-month period was selected to allow completion of any preoperative chemo- and/or radiotherapy, surgery, and postoperative recovery.

The second measure requires the use of radiation therapy for patients younger than age 80 with T4N0M0 or stage III rectal cancer. In the NICCQ study, adherence on a similar measure ranged from 58% to 92% across the five metropolitan areas.⁸ The evidence base supporting this measure demonstrates that it is patients with tumors below the peritoneal reflection who benefit from radiation.^23,24 However, it is not straightforward to identify these patients from tumor registry data because registries classify tumors on either side of the peritoneal reflection as “rectosigmoid.” Therefore, in order to implement a feasible measure and to ensure that only patients for whom radiation is appropriate are evaluated on the measure, the panel chose to limit the denominator to true rectal cancer, excluding all patients with rectosigmoid cancer.

The third colorectal measure assesses whether adjuvant chemotherapy is initiated within 4 months of diagnosis of stage III colon cancer in patients age 80 or younger. In the NICCQ data, patients with high-risk stage II and stage III were not distinguished; the proportion of patients in the combined group who received adjuvant chemotherapy ranged from 78% to 100%.⁸ Because the appropriate use of adjuvant chemotherapy in stage II colon cancer has not been definitively established, the panels limited the ASCO/NCCN measure to patients with stage III cancer.^25,26 The time frame for initiating treatment was selected to allow completion of surgery and recovery from surgical complications.

The final measure applies to colon cancer only, and assesses whether patients with stage II or III disease, who have not undergone preoperative chemotherapy or radiation, and who have 12 or more lymph nodes removed and examined. This measure was proposed by the multidisciplinary content panel based on several observational studies suggesting a survival benefit from removal of at least 12 nodes.^27-31 The value on outcome of removing and examining nodes has not been assessed in a randomized trial. Therefore, the panels agreed to include it as a surveillance measure to foster collection of population-based data on variation in number of nodes removed and examined. The measure excludes patients with rectal cancer (International Classifications of Disease-for Oncology, Third Edition, site code C20.0) because of the variability of the number of nodes in the mesorectum, and because neoadjuvant therapy, commonly used in patients with rectal cancer, may reduce the number of lymph nodes identified pathologically.

DISCUSSION

The ASCO/NCCN Quality Measures are the result of a collaborative effort to answer the call for evidence-based rigorously developed, scientifically sound measures that can be used in national accountability programs dedicated to improving the quality of cancer care. Several features of these measures are noteworthy. The final measures were derived from measure sets from NICCQ and NCCN that have undergone extensive testing and validation supplemented by nationally accepted practice guidelines. Furthermore, prioritization of candidate measures used an explicit, data-driven process designed to ensure that measure adoption would have the maximal beneficial impact on cancer outcomes in the population. We therefore believe that these measures represent a valuable contribution to the growing library of clinical measures supported by national organizations such as the NQF and specialty groups.

Implementation is a major hurdle for any new measure set, including the ASCO/NCCN quality project. The simplest measures to implement would come from codes embedded in the billing system to define the numerator and denominator populations. However, all of the ASCO/NCCN Quality Measures require not only ICD-9 coded information like cancer diagnosis, but also information including stage, estrogen receptor status, tumor size, and node numbers, none of which are captured in claims-based systems. These data are available in the medical record, the source used by both the NICCQ and NCCN projects. However, such detailed review of medical records on a national scale would be cumbersome and expensive unless properly coded electronic health records were widely and successfully implemented throughout the United States.³²

To overcome the limited clinical detail of health care claims, linkages between administrative data and tumor registries have proven rich sources of data for quality of care studies. However, the most widely used linkage, between the National Cancer Institute's Surveillance, Epidemiology and End Results (SEER) program and Medicare claims, provides information on only those patients who live in SEER regions who are covered by Medicare (mostly age 65 or older).³³ Further, the data are often several years old before they are available for analysis.

Therefore, the panel specified the measures using the data dictionary used by hospital tumor registries approved by the Commission on Cancer of the ACoS.³⁴ The data dictionary, termed the Facility Operations and Registry Data Standards Manual is used by ACoS-approved registries that collectively include approximately 70% of cancers treated in the United States. Components of this data dictionary are used by population registries including SEER and state-based registries that submit data to the Centers for Disease Control and Prevention (CDC).

All these registries assemble data from multiple sources including both hospital and outpatient sites. However, all have limitations and quality is variable, especially among the hospital-based tumor registries.^35-37 The most important limitation of registry data is the under-reporting of treatments administered in nonhospital-based outpatient settings. This particularly affects reporting of chemotherapy and radiotherapy.^36-38 So although the data elements required for the ASCO/NCCN measures are all registry-required fields, it must be acknowledged that complete reliance on registry data as currently operationalized will not provide an accurate assessment of performance on measures involving either chemotherapy or radiotherapy. This problem could be ameliorated by an aggressive effort to ascertain outpatient records, but to date, there has been little incentive for registries to invest in such a resource-intensive effort. Specific attention directed to key quality measures, with accountability for performance at the hospital level, would likely lead to a greater willingness to invest resources in the collection of complete and accurate outpatient data, and similarly of complete diagnostic information available only in the outpatient setting. In one study, the CoC found that feedback to hospitals of data on their performance on one of these measures— chemotherapy with stage III colon cancer—prompted more complete reporting of chemotherapy data.³⁹ Ultimately, if payors and professionals value the use of quality measures for accountability, they must work together to develop strategies for sustainable financing of measurement and reporting programs.

Relying on tumor registries rather than the medical record restricted the specifications of some of the measures. For example, though it might be advantageous to designate the specific chemotherapy regimens that constitute quality care for a given cancer, the measures specify only that the patient receive chemotherapy or hormone therapy as this is what is collected by the registry. Future measures may assess the specific drugs given, but in requiring this more granular data, it will need to be recognized that this may place a substantial increased burden for data collection.

The ASCO/NCCN panels carefully considered whether to incorporate provisions allowing exceptions to the measures based on physician recommendations that were not followed, patient preferences against the measure-specified intervention, or important comorbid conditions. Ultimately, the panels decided not to include such exceptions, and to define care as adherent to the measure only when measure recommended interventions were actually administered. Several factors played into this decision. First, an effort to collect information from medical records regarding treatment recommendations was considered to be overly labor intensive, and the validity of this type of data has not been clearly established. Second, allowing documentation of a recommendation to meet the standards for accountability could lead a few physicians to bias information recorded in the medical record. Finally, it is not clear that providers should not be held accountable for unusually high rates of patient refusal of life-saving therapies. Consideration was given to the development of clinically detailed exclusion criteria, but the chart review required to collect this information was felt to be infeasible. Instead, the panels chose the practical strategy of placing upper age limits on many of the measures, excluding patients most likely to have sufficient reasons, such as serious comorbidity, to forego treatment and least likely to experience substantial life expectancy gains. It should be clear that the panels did not intend to suggest that these treatments are inappropriate for all older people; rather, the exclusion will reduce the capture of clinically appropriate variation.

Although it is difficult to predict how the complex forces currently working to reshape the American health care system will play out, it seems likely that payment will increasingly be linked to performance. To date, pay-for-performance approaches have largely been implemented at the health plan or hospital level; but some payers are eager to move this incentive down to the level of the individual physician. The ASCO/NCCN panels considered but ultimately decided against developing physician-targeted measures, focusing instead on measures appropriate for evaluation of hospitals or health care systems. A primary reason for this decision is that within 1 year, individual doctors simply do not treat a sufficient number of patients with specific cancer diagnoses to enable statistically precise comparisons among peers.⁴⁰ Furthermore, cancer care is increasingly multidisciplinary and responsibility for quality is shared among groups of physicians. For example, a patient may not receive radiation after breast conservation because the surgeon does not refer the patient to a radiation oncologist; the referral is made but the hand-off is not completed; the radiation therapist does not recommend or administer radiation; or the patient chooses to forego radiation. Every step in the process must be successful for the patient to receive the intervention assessed on that measure. The methodology for allocating responsibility and payment among participants for achieving success on such a quality measure has not been developed at this time. Similarly, both the surgeon and pathologist share responsibility for assuring that 12 or more lymph nodes are removed and examined, but billing and payment systems currently lack the capacity to recognize this joint responsibility. Despite these concerns, other groups are proceeding to use similar measures for individual physician accountability, including most notably the Physician Quality Reporting Initiative implemented by CMS in July 2007.⁴¹

Other organizations also have focused on measurement of the quality of cancer care. Most notable is the NQF that established a formal project for evaluation and approval of quality measures for breast cancer, colorectal cancer, and palliative care. The program was funded by several federal agencies including the Agency for Healthcare Research and Quality, National Cancer Institute, CMS, and the CDC. The NQF used a different method to identify measures than the ASCO/NCCN group. Briefly, in 2004 the NQF issued a national call for measures developed by other organizations and also used evidence-based reviews of quality measures commissioned by the NCI. Measures identified by both methods were then reviewed by technical panels and an NQF Steering Committee, and in some cases referred back to the proposing organization for refinement. Many of the cancer measures recently endorsed by the NQF were proposed by the CoC of the ACoS based on data from the National Cancer Data Base. These measures are also intended for use as accountability and pay-for-performance measures at the hospital or health system level. Of importance is that the draft NQF measures proposed for breast and colorectal cancer were similar to those identified by the ASCO/NCCN panels. After release of the ASCO/NCCN measures in August 2006 (and before the completion of the NQF review process), the ASCO/NCCN and CoC/NQF groups collaborated to reconcile differences in the specifications for the measures to assure a uniform set of standards for use in the United States. The methods and results of this final harmonization of the common measures and the final measures approved by the NQF are described in Appendix 1.

It is expected that these measures will enter practice in nationwide quality improvement programs. To date we are aware of two such efforts. The first is incorporation of these measures into ASCO's Quality Oncology Practice Initiative.^40,42 This program collects data from retrospective chart reviews conducted in the outpatient offices of participating medical oncologists. Data are submitted to a central ASCO database and analyzed to create practice-specific reports for quality improvement. Several of the ASCO/NCCN/NQF measures appropriate for assessing care provided by the medical oncologist were adapted for office-based analysis. The second program is the Electronic Quality Information Packet program of the CoC.⁴³ The ASCO/NCCN/NQF measures are applied to data in the National Cancer Data Base for each participating cancer program. The level of concordance with each measure and the patient level data are provided to each cancer program. The CoC expects to extend this program to other cancer measures, and to use performance on these measures as a component of CoC cancer program approval.

These quality measures will likely require modifications in the future, based on changes in guideline recommendations and data on application of these measures. An example of new data that may prompt measure review are two recent studies of the impact of colon cancer lymph node counts on outcome. One based on SEER-Medicare data⁴⁴ concluded that lymph node count may not be a useful quality measure.⁴⁴ The other, a systematic review of published data, showed a survival advantage with high node counts in stage II and III cancer.⁴⁵ These and other data will be used as this measure is reviewed by the ASCO/NCCN group, and may lead either to re-endorsement or change in the measure. The ASCO/NCCN collaborative group is committed to periodic review and updating of all these measures to assure their relevance to current clinical practice.

In conclusion, ASCO and NCCN used available data and experts to collaborate in development of a set of three breast and four colorectal quality measures for application at the hospital or health system level. After release of these measures, the ASCO/NCCN group worked with the CoC/NQF to harmonize the measures from both groups to establish a single nationally approved set of quality measures.

AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Although all authors completed the disclosure declaration, the following author(s) indicated a financial or other interest that is relevant to the subject matter under consideration in this article. Certain relationships marked with a “U” are those for which no compensation was received; those relationships marked with a “C” were compensated. For a detailed description of the disclosure categories, or for more information about ASCO's conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors.

Employment or Leadership Position: Joan McClure, National Comprehensive Cancer Network (C); Stephen B. Edge, American College of Surgeons (U), National Comprehensive Cancer Network (U) Consultant or Advisory Role: None Stock Ownership: None Honoraria: Eric C. Schneider, ASCO Research Funding: Eric C. Schneider, ASCO; Katherine L. Kahn, Recieved research support as co-PI of NICCQ study described in this article Expert Testimony: None Other Remuneration: None

AUTHOR CONTRIBUTIONS

Conception and design: Christopher E. Desch, Kristen K. McNiff, Deborah Schrag, Joan McClure, Eva Lepisto, Molla S. Donaldson, Katherine L. Kahn, Jane C. Weeks, Stephen B. Edge

Administrative support: Kristen K. McNiff

Provision of study materials or patients: Eva Lepisto, Katherine L. Kahn

Collection and assembly of data: Christopher E. Desch, Kristen K. McNiff, Eric C. Schneider, Deborah Schrag, Eva Lepisto, Katherine L. Kahn, Jane C. Weeks, Clifford Y. Ko, Andrew K. Stewart, Stephen B. Edge

Data analysis and interpretation: Eric C. Schneider, Deborah Schrag, Joan McClure, Eva Lepisto, Molla S. Donaldson, Katherine L. Kahn, Jane C. Weeks, Clifford Y. Ko, Andrew K. Stewart, Stephen B. Edge

Manuscript writing: Christopher E. Desch, Kristen K. McNiff, Eric C. Schneider, Deborah Schrag, Eva Lepisto, Molla S. Donaldson, Katherine L. Kahn, Jane C. Weeks, Stephen B. Edge

Final approval of manuscript: Kristen K. McNiff, Eric C. Schneider, Deborah Schrag, Joan McClure, Eva Lepisto, Molla S. Donaldson, Katherine L. Kahn, Jane C. Weeks, Clifford Y. Ko, Andrew K. Stewart, Stephen B. Edge

Appendix 1: Harmonization of ASCO/NCCN Measures With NQF Measures

ASCO and NCCN developed the quality measures reported in this article between January and August 2006 and released the measures on their Web sites in August 2006. The American College of Surgeons’ (ACoS) Commission on Cancer (CoC) also developed quality measures for breast and colorectal cancer over the past 2 years, and submitted the measures for National Quality Forum (NQF) endorsement using a process detailed by the NQF.⁴⁶ Briefly, the NQF solicited candidate quality measures in a national call. Candidate measures were reviewed by multidisciplinary technical panels for each disease site. Measures selected by the NQF for further validation, revision, and approval met similar standards as those used in the ASCO/NCCN process including clinical relevance, known variation in practice, and applicability using existing data resources. The process measures submitted by the CoC underwent further validation and revision based on data in the National Cancer Data Base and recommendations from the NQF panels and Steering Committee, with support from the National Cancer Institute. The NQF approved these measures for assessment of the treatment of breast and colorectal cancer in April 2007.

The measures reviewed by the NQF were similar to six of those developed by the ASCO/NCCN process (ASCO/NCCN put forth one measure on use of chemotherapy after surgery for rectal cancer, a process not addressed by NQF). ASCO, NCCN, and the CoC sought to determine if the differences in measure specifications could be reconciled so that a single set of measures would be provided to end users. In November 2006, representatives of each organization met and made minor changes in the measure specifications as detailed below. The ASCO/NCCN collaborative group updated the specifications of the measures on their Web sites. The harmonized measures were those submitted for final review by the NQF Steering Committee and were those approved in April 2007 for use nationwide. Appendix Table A1 shows the final CoC/ASCO/NCCN measures as endorsed in April 2007 by the NQF (Table A1 also includes the ASCO/NCCN rectal cancer measure not approved by NQF).

The most significant issue requiring harmonization was whether to include as concordant care situations in which the appropriate care was considered by the physician but not administered or declined by the patient. The ASCO/NCCN panels did not include these cases as concordant due to concerns that these determinations are subjective and subject to provider bias in reporting that could affect the validity of quality monitoring programs. In addition, making the determination that care was “considered” requires subjective interpretation of the medical record, making consistent data across practice settings difficult to obtain.

The CoC and NQF included care as “concordant” where treated was recommended but not administered, or omitted for valid medical reasons (eg, comorbidity) because it more accurately represents clinical practice, and was therefore considered more reasonable for provider accountability. Counting these cases as nonconcordant care could disproportionately affect some providers unless the proportion of cases meeting such exclusions is equal between providers and health care systems. In addition, experts on these panels were concerned physicians might feel compelled to overtreat some patients in order to maximize reported compliance. The cancer registry system includes fields for collection of the reasons for omission of treatment, and registrars collect this information whenever possible from the medical record without apparent undue burden.

The reconciliation group agreed to address this issue by specifying that three numbers be reported for each measure: the percentage of patients who receive the specified treatment, the percentage for whom the treatment was reported as considered but not administered, and the sum of the two.

The other differences between the measures were easily reconciled without compromising the intent of the measure or the data supporting that measure.

For a number of measures, ASCO/NCCN specifically excluded women younger than age 18. This exclusion was not explicitly stated in CoC measures. This exclusion was applied to all the measures.

For chemotherapy for women younger than age 70 with estrogen receptor–negative cancer, ASCO/NCCN excluded women with T1cN0M0 cancer (1.1 to 2.0 cm, negative nodes). Based on the data on treatment patterns in NICCQ, NCCN, and the National Cancer Data Base, and on the level of expected benefit for women younger than age 70, the reconciliation panel included these cases in the final measure.

For breast cancer chemotherapy, ASCO/NCCN specified, “adjuvant chemotherapy” and CoC specified, “combination chemotherapy”. The term in the final measure is, “adjuvant combination chemotherapy.”

The allowed time frames differed in a number of the measures and were reconciled as presented in Table 2.

ASCO/NCCN included in their measure set a measure that was not included by in the CoC set, evaluating the use of chemotherapy after surgery in rectal cancer. This measure was not addressed by the reconciliation panel and was not considered by the NQF.

The reconciled measures were the final set put forward by the NQF for final approval, which occurred in April 2007. The final measures are presented in Table 2 and are available at the Web sites of the CoC, NQF, NCCN, and ASCO.

Appendix 2: ASCO and NCCN Content and Methodology Panelists