Incorporation of Core Competency Questions Into an Annual National Self-Assessment Examination for Residents in Physical Medicine and Rehabilitation: Results and Implications

Webster, Joseph B.

doi:10.1016/j.pmrj.2008.11.008

« Previous Next »PM&R
Volume 1, Issue 3 , Pages 229-233, March 2009

Incorporation of Core Competency Questions Into an Annual National Self-Assessment Examination for Residents in Physical Medicine and Rehabilitation: Results and Implications

Division of PM&R, University of Utah, 30 North 1900 East, Salt Lake City, UT, American Academy of PM&R Self-Assessment Committee^†

Received 22 July 2008; accepted 15 November 2008.

Objective

To determine the performance and change over time when incorporating questions in the core competency domains of practice-based learning and improvement (PBLI), systems-based practice (SBP), and professionalism (PROF) into the national PM&R Self-Assessment Examination for Residents (SAER).

Design

Prospective, longitudinal analysis.

Setting

The national Self-Assessment Examination for Residents (SAER) in Physical Medicine and Rehabilitation, which is administered annually.

Participants

Approximately 1100 PM&R residents who take the examination annually.

Interventions

Inclusion of progressively more challenging questions in the core competency domains of PBLI, SBP, and PROF.

Main Outcome Measurements

Individual test item level of difficulty (P value) and discrimination (point biserial index).

Results

Compared with the overall test, questions in the subtopic areas of PBLI, SBP, and PROF were relatively easier and less discriminating (correlation of resident performance on these domains compared with that on the total test). These differences became smaller during the 3-year time period. The difficulty level of the questions in each of the subtopic domains was raised during the 3 year period to a level close to the overall exam. Discrimination of the test items improved or remained stable.

Conclusions

This study demonstrates that, with careful item writing and review, multiple-choice items in the PBLI, SBP, and PROF domains can be successfully incorporated into an annual, national self-assessment examination for residents. The addition of these questions had value in assessing competency while not compromising the overall validity and reliability of the exam. It is yet to be determined if resident performance on these questions corresponds to performance on other measures of competency in the areas of PBLI, SBP, and PROF.