Using Item Response Theory to Evaluate the Quality of Multiple-Choice Questions

Abstract

The quality of multiple-choice questions (MCQs) has traditionally been evaluated using Classical Test Theory (CTT), which relies on indices such as item difficulty, item discrimination, and test reliability. Although CTT is simple, widely used, and easily implemented with conventional statistical software, it has important limitations. Item characteristics are dependent on the examinee sample, whereas estimates of examinee ability depend on the specific test administered, making comparisons across different cohorts or test forms difficult. Furthermore, CTT provides only a single summary statistic for each item and does not describe how item performance varies across different ability levels.

Item Response Theory (IRT) was developed to address these limitations by modeling the relationship between an examinee's latent ability (θ) and the probability of answering an item correctly. Since its development by Frederic M. Lord, Georg Rasch, and Allan Birnbaum, IRT has become the dominant psychometric framework for educational and professional testing. Unlike CTT, IRT estimates item parameters—including difficulty, discrimination, and guessing—independently of the examinee sample, provided that model assumptions are satisfied. It also provides detailed information on how individual items function across the ability continuum. These advantages have made IRT the foundation of modern assessment practices, including item banking, computerized adaptive testing, and test equating. Although IRT is more mathematically sophisticated than CTT, recent advances in statistical software such as Stata and R have made its application increasingly accessible.

This review summarizes the fundamental concepts of IRT, explains the interpretation of its key parameters, and discusses its practical application in evaluating the quality of MCQs in medical education.