Study reveals benefits of repeated training for medical educators' exam question development
A recent study conducted by researchers at Pusan National University highlights the value of repeated item development training for medical school faculty.
The study reveals that such training significantly enhances educators' ability to predict and adjust the difficulty of exam questions, thereby improving assessment accuracy and educational outcomes.
The research, published in BMC Medical Education, involved comprehensive item development workshops with 62 faculty members conducted in 2016 and 2018. Participants received continuous feedback and training to refine their question development skills.
The study compared predicted difficulty levels of multiple-choice questions (MCQs) with actual student performance data. Initial findings showed significant alignment in one subject (cardiology) before training, which expanded to four subjects (cardiology, neurology, internal medicine, and preventative medicine) post-training.
Evaluation plays a critical role in education, particularly in medical training where precise assessment of students' knowledge is vital. The item difficulty index, which measures the proportion of students correctly answering a question, is central to this process. Accurate prediction of item difficulty ensures that assessments are fair and effective.
Educators often face challenges in accurately predicting item difficulty, which can lead to overestimations. The study highlights the limited research on faculty development programs aimed at improving these predictive skills.
To address this gap, the research team led by Professor Sang Lee, Vice President for Medical Affairs, conducted targeted training to enhance faculty abilities in this area.
The workshops involved drafting, reviewing, and revising exam questions to meet national exam standards, focusing on application-based questions within an ideal difficulty range.
Continuous feedback was provided by an item development committee, and the accuracy of faculty predictions was compared with fourth-year medical student performance.
The study demonstrated that repeated training significantly improves faculty members' ability to predict and adjust item difficulty, leading to more effective assessments and better educational outcomes.
Professor Lee, noted:
“Repeated item development training not only helps adjust the difficulty level but also enhances the construction of the items, increases their discriminating power, and properly addresses the issue of validity.
“Soon there will be an era of item development using AI. For that, studies like ours are important for providing necessary information about existing items and students' answer data, which will help in developing an AI-powered automated item development programme.”
