This event is being organized by the NCME Artificial Intelligence in Measurement and Education (AIME) SIGIMIE.
Effective educational measurement relies heavily on the curation of well-designed item pools. However, item calibration is time consuming and costly, requiring a sufficient number of respondents to estimate the psychometric properties of items.
In this presentation, based on several recently published papers, we explore the potential of six different large language models (LLMs; GPT-3.5, GPT-4, Llama 2, Llama 3, Gemini-Pro and Cohere Command R Plus) to generate responses with psychometric properties comparable to those of human respondents. Results indicate that some LLMs exhibit proficiency in College Algebra that is similar to or exceeds that of college students. However, we find the LLMs used in this study to have narrow proficiency distributions, limiting their ability to fully mimic the variability observed in human respondents, but that a novel ensemble of LLMs can better approximate the broader ability distribution typical of college students. Utilizing item response theory, the item parameters calibrated by LLM respondents have high correlations (eg, >0.8 for GPT-3.5) with their human calibrated counterparts. Several augmentation strategies are evaluated for their relative performance, with resampling methods proving most effective, enhancing the Spearman correlation from 0.89 (human only) to 0.93 (augmented human).
Additionally, in this talk, we will report on related published research which finds that LLMs have the capability to produce Algebra items with similar psychometric properties as subject matter expert-authored items. Finally, we will discuss the generalizability and relationship of these findings to both summative and formative assessment contexts.
Presenters:
- Yunting Liu, Shreya Bhandari, and Zachary Pardos from UC Berkeley School of Education