Residency evaluations. Are they worth the effort?

A. L. Scheuneman, J. P. Carley and W. H. Baker
Department of Surgery, Loyola University Medical Center, Maywood, IL.

OBJECTIVE: The accuracy of rotation evaluation forms was assessed to determine the usefulness of clinical performance ratings in predicting competence in a general surgery residency. DESIGN: Longitudinal, 15-year, multifactorial, repeated-measures design. SETTING: University medical center residency training program. PARTICIPANTS: General surgery and subspecialty residents (n = 310) entering postgraduate training from 1976 to 1990 received 6459 rotation evaluations. Predictive efficiency of 4423 evaluations of a subsample of 199 residents admitted to the general surgery program during the study period was assessed using several outcome measures. MAIN OUTCOME MEASURE: Attending surgeons categorized and ranked residents' competence at the completion of training. RESULTS: Rotation evaluations identified 77.8% of residents who involuntarily withdrew. Prediction of outcome category (problem, average, or superior residents), as well as final ranking (R2 = .54), were demonstrated (P < .001). Evaluations of residents from the first year of training onward predicted outcome at the end of training (P < .01). CONCLUSION: Rotation evaluations demonstrate predictive validity in documenting resident performance, particularly when the accuracy of individual raters is known.