This event is being organized by the NCME Artificial Intelligence in Measurement and Education (AIME) SIGIMIE.
Finding Words Associated with DIF: Predicting Differential Item Functioning using LLMs and Explainable AI
We fine-tuned and compared several encoder-based Transformer large language models to predict differential item functioning (DIF) from the item text. We then applied explainable AI methods to these models to identify specific words associated with DIF. The data included 42,180 English language arts and mathematics items. Prediction R2 ranged from .04 to .32 among eight focal and reference group pairs.
Many words associated with DIF reflected minor sub-domains included in the test blueprint by design, rather than construct-irrelevant item content that should be removed from assessments. This may explain why qualitative reviews of DIF items often yield confusing or inconclusive results.
Our approach can be used to screen words associated with DIF during the item-writing process for immediate revision or help review traditional DIF analysis results by highlighting key words in the text. Extensions of this research can enhance the fairness of assessment programs, especially those that lack resources to build high-quality items, and among smaller subpopulations where we do not have sufficient sample sizes for traditional DIF analyses.
Presenter:
National Council on Measurement in Education