Utility of 360-degree assessment of residents in a Turkish academic emergency medicine residency program

Abstract

Objectives: This study was designed to test a 360-degree assessment tool for four of the emergency medicine resident competencies as outlined by the Council of Residency Directors in Emergency Medicine on patient care, communication skills, professionalism and system based practice in an academic Emergency Department.

Material and methods: Using the competency framework of the American Accreditation Council for Graduate Medical Education, a 57 item-containing assessment tool was created. Based on the different exposure aspects of the involved evaluator groups, the items were integrated into seven different evaluation forms. All sixteen of 16 residents and members from each evaluator group voluntarily participated in the study. Internal consistency scores, multilayer and multilevel Kappa values were measured. Evaluator group scores and resident ranks in competency areas were compared. All evaluators were asked to comment on the applicability and usefulness of the assessment tool in emergency medicine.

Results: Seven groups completed a total of 1088 forms to evaluate 16 residents. The reliability coefficient for the faculty members was 0.99 while it was 0.60 for the ancillary staff. The interrater Kappa values for faculty members, nurses and peer assessment were relevant with a value of greater than 70%.

Discussion and conclusion: Our results showed that the 360-degree assessment did meet expectations by the evaluator group and residents, and that this method was readily accepted in the setting of a Akdeniz University Emergency Medicine residency training program. However, only evaluations by faculty, nurses, self and peers were reliable to have any value. Doing a 360° evaluation is time and effort consuming and thus may not be an ideal tool for larger programs.