The conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in "real world" settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.

To provide this information, comparative effectiveness research must assess a comprehensive array of health-related outcomes for diverse patient populations and subgroups. Defined interventions compared may include medications, procedures, medical and assistive devices and technologies, diagnostic testing, behavioral change, and delivery system strategies. This research necessitates the development, expansion, and use of a variety of data sources and methods to assess comparative effectiveness and actively disseminate the results.

A variety of research methods are appropriate for addressing these research questions, such as clinical trials (including "pragmatic" trials that assess treatment outcomes in typical real-world settings), randomized or cluster-randomized health services studies, observational studies with methods that account for confounding, analysis of secondary data using tools such as disease registries or linked administrative health data sets, and synthetic studies such as systematic reviews, meta-analyses, and analytical modeling studies. An Institute of Medicine report² on comparative effectiveness research recommended a variety of strategic directions for long-term investment in comparative effectiveness research, including ensuring meaningful consumer, patient, and caregiver participation; building robust information systems and research methods; development and support of a highly skilled comparative effectiveness research workforce; and supporting efforts to translate comparative effectiveness research knowledge into everyday clinical practice.

WHAT ARE THE CHALLENGES FOR COMPARATIVE EFFECTIVENESS RESEARCH OF SURGICAL PROCEDURES?
There are a variety of impediments to the evaluation of surgical procedures. Many of these have received a large amount of attention, such as the reluctance of patients to enroll in randomized trials, difficulty with blinding and standardization, ethical concerns, and lack of equipoise among surgeons. However, there are also a number of fundamental reasons why there is so little high-quality surgical research.

The regulatory environment for procedures and devices is probably the single biggest barrier to rigorous evaluation of surgical procedures. Because federal statutes mandate evidence from clinical trials before a new drug can be marketed, there is an entire industry to support clinical trials of pharmacologic agents. The type of evidence required to market a new device, either no evidence (through the Food and Drug Administration's 510[k] pathway) or rudimentary clinical data from small open-label studies in the premarket notification pathway, effectively dictates that no natural sponsor of rigorous evaluation of devices will ever emerge. Even worse, surgical procedures that are not predicated on the use of a device are not subject to any central regulatory framework.

If there are legitimate reasons why we have so few randomized trials of surgical interventions, why is it that so much of the rest of surgical research is done so poorly? The vast majority of studies of surgical procedures are uncontrolled case series and nonrandomized comparative studies. In general, these studies are so susceptible to selection bias and measurement biases that they cannot provide any valid information on treatment effectiveness. Part of the answer to the question of why there are so many poor-quality studies lies in the type of information that can be derived from case series, principally, the fact that a particular procedure can actually be done (often called "feasibility") and an estimate of the likely consequences of the procedure in a particular group of patients in whom it was performed. While this type of information may be highly valuable to surgeons, it is unfortunately often mistaken for evidence of effectiveness, acceptability, or some other measure of the treatment's value.

Comparative nonrandomized studies of surgical procedures are highly problematic. Selection bias is a major, and often unquantifiable, source of error. Retrospectively studying groups of patients treated by 1 procedure or another, while intuitively appealing, is a major methodological flaw. For example, a study of patients with pancreatic cancer who had a Whipple procedure is unlikely to produce information that is truly applicable to other surgeons or other centers. Since relatively few patients with pancreatic cancer are treated with a Whipple procedure, the process of selection of patients for surgery has a major influence on the outcomes of the patients who are treated. Surgeons are highly skilled at patient selection—the "art" of surgical judgment has much to do with deciding which patients should not have surgery—and no statistical method can adequately adjust for the effects of such profound selection biases. A much better approach would be to study a group of patients defined by a disease or condition rather than the surgical procedure (for example, patients with pancreatic cancer, rather than patients who had a Whipple procedure).

To overcome the limitations of selection biases in nonrandomized studies of surgical procedures, study populations must include larger denominators, such as subjects with a disease or condition, rather than only those patients who actually received a surgical procedure. There is a need for prospective studies of patients who can be treated by particular surgical interventions, rather than retrospective studies of patients who were treated by a particular procedure.

EVIDENCE-BASED POLICY MAKING
Comparative effectiveness research is currently being used in some jurisdictions to make policy and coverage decisions. At the symposium, the case of computed tomographic (CT) colonography (virtual colonoscopy) as a colorectal cancer screening test was reviewed from the perspective of policy making in 3 different health jurisdictions: Medicare Evidence Development and Coverage Advisory Committee, Ontario Health Technology Advisory Committee, and Washington State Health Care Authority/Institute for Clinical and Economic Review. While policy making and coverage decisions are increasingly made on the basis of research evidence, the evidence on which decisions are made is typically very weak. Screening tests pose a particular problem for technology evaluation, because it is so difficult to estimate their effect on treatment outcomes. The challenge of estimating a survival benefit of screening is compounded by potential harms associated with false-positive results, false-negative results, overdiagnosis, and cost increase without an outcome benefit.

As a rule, the most important characteristic of a cancer screening test is its negative predictive value. A high negative predictive value indicates that a negative test result reliably predicts a lesion-free state; thus, a more invasive diagnostic test may be avoided. Based on a 97% to 99% negative predictive value for polyps more than 6 mm,^3-4 average-risk individuals 50 years and older with normal CT colonography results may safely forego (conventional) optical colonoscopy with its inherent risks.

However, the criteria for comparative effectiveness research include not only a head-to-head comparison, but also assessment of the net benefit, that is benefits and harms, and the effect in a patient-centered or real-world setting. Appropriate measures of potential harms of CT colonography (exposure to radiation, management of incidental extracolonic findings) are absent currently; therefore, net benefit cannot be calculated. Perhaps more importantly, the prevalence and behavior of polyps among Medicare recipients are different from average-risk individuals aged 50 years and older; thus, test characteristics specific to the Medicare population are unknown. Ultimately, in spite of excellent test characteristics compared with optical colonoscopy, the Centers for Medicare & Medicaid Services declined to cover CT colonography for these reasons. Similar concerns regarding sampling were advanced in Ontario, Canada, and led to a decline of coverage, and similar concerns regarding test characteristics were expressed by the Washington State Department of Health. It is conceivable that when comparative effectiveness research fulfills the additional criteria for assessment, coverage may be expanded to include this new technology.

PUTTING IT ALL TOGETHER
The symposium on surgical comparative effectiveness research highlighted key directions and opportunities for this area of inquiry. Studies of surgical procedures must move away from the popular but unproductive "surgical series" and nonrandomized comparative studies and move toward a paradigm of studying population-based samples of patients with diseases. The opportunities for using research to inform evidence-based policy have never been greater.

An agenda for surgical comparative effectiveness research is beginning to take shape. Legislative, regulatory, and funding agencies are creating a favorable environment for research on the real-world effectiveness of surgical procedures, and there is an increasing opportunity for comparative effectiveness research to impact health care, health policy, and perhaps the sustainability of the health care system. Surgeons must continue to lead in this area of research.

AUTHOR INFORMATION
Correspondence: Dr Morris, Department of Surgery, University of Michigan, 211 N 4th Ave, Ste 2A, Ann Arbor, MI 48104 (ammsurg{at}med.umich.edu).

Author Contributions: Study concept and design: Urbach and Morris. Acquisition of data: Urbach and Morris. Analysis and interpretation of data: Urbach and Morris. Drafting of the manuscript: Urbach and Morris. Critical revision of the manuscript for important intellectual content: Urbach and Morris. Administrative, technical, and material support: Morris. Study supervision: Morris.

Financial Disclosure: None reported.