Development of a Competency-based Assessment System for Physicians-in-Training

June 5, 2017 10:40 AM – 11:30 AM

Patricia Hicks, Children's Hospital of Philadelphia and the Perelman School of Medicine at the University of Pennsylvania
Alan Schwartz, Department of Medical Education, University of Illinois at Chicago
Melissa Margolis, National Board of Medical Examiners
Brian Clauser, National Board of Medical Examiners
Carol Carraccio, American Board of Pediatrics
Martin Bruegel, National Board of Medical Examiners

Background: The medical community needs a competency-based assessment system (CBAS) that provides: easy (mobile) access; efficient rater scoring of relevant content; and, reports that provide meaningful assembly and display of assessment data to all end-users.

Objective: We have developed a CBAS that provides evidence of Pediatric intern performance in the inpatient setting that can inform learners and those responsible for learner advancement about their readiness to serve without a supervisor nearby.

Design/Methods: A mobile-entry CBAS was implemented in 15 Pediatrics residency training programs. Multisource feedback (MSF) and structured clinical observation (SCO) data were collected by rotation; end-of-rotation and clinical competency committee (CCC) reports included competency-level scores, recommendations for improvement, and general comments. Outcomes were analyzed to evaluate the overall utility of the system with emphasis on reliability, validity, acceptability, and educational impact.

Results: An average of 5 MSF and 3.7 SCO instruments were completed across the 524 participating residents; instruments required 4 - 8 minutes to complete. Generalizability coefficients over 0.80 were attainable with six MSF observations by residents, faculty, or nurses. Most observers, interns, program directors, and clinical (CCC) members indicated that the new system added value to their existing assessment program. Interns reported improved quality, quantity, frequency, specificity, and timeliness of feedback. CCC feedback reports led to increased confidence in CCC milestone assignments. Participating sites reported no difficulties with functionality. The PMAC CBAS has demonstrated the ease of use, validity evidence and acceptability called for in the busy clinical workplace.