National Board of Medical Examiners Internship

Summer 2021 Internships in Assessment Science and Psychometrics

June 7 - July 30, 2021   Philadelphia, PA

This year’s internship will be virtual, due to uncertainty surrounding the Covid-19 pandemic. 

Interns will interact with other graduate students and NBME staff and will present completed projects or work-in-progress to NBME staff. Internships typically result in conference presentations (e.g., NCME) and sometimes lead to publication or dissertation topics. 

Requirements 

• Active enrollment in doctoral program in measurement, statistics, cognitive science, medical education, or related field; completion of two or more years of graduate coursework.
• Experience or coursework in one or more of the following: test development, IRT, CTT, statistics, research design, and cognitive science. Advanced knowledge of topics such as equating, generalizability theory, or Bayesian methodology is helpful. Skill in writing and presenting research. Working knowledge of statistical software (e.g., Winsteps, BILOG; SPSS, SAS, or R).
• Interns will be assigned to one or more mentors but must be able to work independently.
• Must be authorized to work in the US for any employer. If selected, F-1 holders will need to apply for Curricular Practical Training authorization through their school’s international student office and have a social security number for payroll purposes. 

Compensation 

Total compensation for the two months is approximately $8000. 

Research Projects 

Interns will help define a research problem; review related studies; conduct data analyses (real and/or simulated data); and write a summary report suitable for presentation. Projects are summarized below. Applicants should identify 2 projects by number that they prefer to work on. 

1. Exploring Response Process Validity Evidence for a Medical Licensing Examination Program. Passing scores on the United States Medical Examination (USMLE) sequence are intended to signal readiness for the unsupervised practice of medicine in the United States. This project will examine response process validity evidence for USMLE score interpretations. USMLE is comprised of three computer-based multiple-choice question examinations: Step 1, Step 2 Clinical Knowledge, and Step 3. For each multiple-choice question on each Step examination, various testing features are available to examinees. These features include the ability to 1) highlight examination text, 2) cross out examination text 3) take notes in a blank window, and 4) view lab values. This project will examine the associations among performing these activities and performance on the examinations. Inferential statistical analyses will be done for all examinees as well as by examinee subgroup, defined by such characteristics as gender and location of medical school. Among other responsibilities, an intern for this project would help to design the study, manage and manipulate a large complex data set, conduct statistical analyses, and interpret and present/describe results. 

2. Virtual Performance Assessment. The United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), in existence since 2004, is undergoing a revitalization. Before the planned rollout of the restructured exam, the NBME will be conducting multiple research studies. Possible projects include work on generalizability, quality assurance metrics, or evaluation of scoring/equating designs for this virtual examination. 

3. What Determines Physician Competence? The USMLE assesses the competence level of future physicians based on the system of tests that check the breadth and depth of medical knowledge, as well as clinical and doctor-patient communication skills. In addition to the USMLE scores, a plethora of exam scores are collected throughout medical school and residency. The goal of this project is to examine the relationships between the various intermediate educational medical tests and the ultimate physician competence indicators. 

4. Interactive Psychometric Dashboards. Contemporary methods of communicating score information are evolving beyond fixed displays on traditional 8 ½ by 11-inch page layouts. Additionally, score users demand more flexibility and specificity to draw tailored inferences in support of evidence-based decisions. For this project the intern will contribute to the development of dashboard products using R Shiny and participate in engagement efforts with internal and external customers. 

5. Using NLP to Explore Item Bias and Test Validity. The use of Natural Language Processing (NLP) within applications, in conjunction with psychometric theory and statistics, has shown great promise for providing innovative information that was not previously available in the item and test development processes of high-stakes licensure programs. This project will focus on the use of basic NLP techniques and application development intended to analyze text in large item banks, to improve item and test development and provide more evidence for test fairness and the validity of test scores. An intern with little or no NLP experience, but who is interested in learning more about this field, may be suitable. Interns with previous NLP experience will have the possibility of incorporating more advanced NLP techniques and analyses into their work, depending on the project scope and other projects underway at NBME. 

6. Measurement Instrument Revision and Development. This project will continue ongoing work on revising a commonly used measurement instrument so that the appropriate inferences can be made about medical student well-being. Duties may include the following: working with subject-matter experts to revise the existing items; conducting think-alouds with medical students; developing a pilot measure of potential items; exploratory and confirmatory factor analysis of initial pilot results to gather structural validity evidence; developing a larger survey to gather concurrent and discriminate validity evidence with the revised measure; and administration and evaluation of the larger survey. 

7. Qualitative Analysis of Focus Group and Interview Data. The RENEW (Re-examining Exams: NBME Effort on Wellness) task force at NBME is focused on understanding the relationship between the pressures of high-stakes licensure examinations and medical student well-being. As part of this work, a series of focus groups and interviews have been completed, but this data has not been exhaustively utilized and analyzed. The intern assigned to this project will apply qualitative and mixed-method methodologies to learn more about the information gathered during the focus groups and interviews. 

8. Computer-Assisted Scoring of Constructed Response Test Items. Relying on subject matter experts to score constructed-response test items is expensive, time-consuming, and introduces natural scoring variations inherent when using human raters. Recently the NBME has developed a computer-assisted scoring program that utilizes natural language processing (NLP) to mitigate these issues. The two main components of the program are (1) ensuring that the information in the constructed response is correctly identified and represented; and (2) building a scoring model based on these concept representations. Current areas of research surrounding this project include (but are not limited to): refining quality control steps to be taken prior to an item being used in computer-assisted scoring; linking and equating computer-assisted scores with human rater scores; evaluating a scoring method based on using orthogonal arrays; and developing metrics that assess item quality and test reliability when computer-assisted scores and human scores are used to make classification decisions. The final project will be determined based on a combination of intern interest and project importance. 

9. Modern Test Construction Methods. The art and science of constructing an efficient and effective high-stakes exam requires balancing several competing aims. These theoretical and statistical considerations are explicitly enumerated when constructing an exam, resulting in a set of constraints that require complex algorithms to build secure test forms. Recent research has suggested that some of these constraints may be unnecessarily restrictive, resulting in test item banks that aren’t efficiently utilized and increased costs for test development and construction. An intern for this project would review current practices for exam construction and examine which constraints may be relaxed while still administering an exam with appropriate domain coverage and exam security considerations. 

Application 

Candidates may apply by going to https://nbme.applicantpro.com/jobs/. A cover letter outlining experience and listing project interests by number, along with a current resume, are required. Application deadline is January 31, 2021. 

All applicants will be notified of selection decisions by February 26, 2021.

Overview 

NBME offers a versatile selection of high-quality assessments and educational services for students, professionals, educators, and institutions dedicated to the evolving needs of medical education and health care. To serve these communities, we collaborate with a comprehensive array of professionals including test developers, academic researchers, scoring experts, 

practicing physicians, medical educators, state medical board members, and public representatives. 

Together with the Federation of State Medical Boards, NBME develops and manages the United States Medical Licensing Examination. In addition, we are committed to meeting the needs of educators and learners globally with assessment products and expert services such as Subject Examinations, Customized Assessment Services, Self-Assessments, the International Foundations of Medicine, and Item-Writing Workshops. 

We also provide medical education funding and mentorship through the Latin America Grants, Stemmler Fund, and Strategic Educators Enhancement Fund, which serve to advance assessment at educators' and health professionals' own institutions. 

NBME employs approximately 30 doctoral level psychometricians and assessment scientists, as well as several MDs specializing in medical education. Staff is recognized internationally for its expertise in statistical analysis, psychometrics, and test development. 

Learn more about NBME at NBME.org. 

Diversity, Equity, and Inclusion: 

At NBME, we continue to innovate and improve how we fulfill the evolving needs of the health care community. This commitment starts and ends with the people at NBME. By recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come. Learn more about NBME at NBME.org. 

Description

If you are a creative and innovative individual who wants to help shape the future of learning and assessment, we encourage you to apply for the 2021 Summer Research and Measurement Sciences (RMS) Internship program. Steeped in decades of broad expertise, RMS conducts rigorous foundational and applied research on the most critical issues facing education and the workforce. Central to ETS’s legacy of global leadership in learning and assessment, RMS is dedicated to advancing the science and practice of measurement, driving innovation in digital assessment, learning and teaching.

Applying for an Internship at RMS

As an intern in RMS, you’ll work with experts who are nationally and internationally known as thought leaders, trusted advisors and go-to collaborators for their high-impact work addressing significant educational and societal goals. ETS staff in RMS have expertise in psychology, education, psychometrics, measurement, statistics, cognitive or learning sciences and data science.

Interns who are accepted into the program will collaborate with scientists on projects related to these topics and will participate in data analysis, writing and other research tasks. Doctoral students who have completed at least two years in one of these or a related field are encouraged to apply. Upon the completion of the program, you’ll have the opportunity to present your findings to teams across R&D.

Note: Applicants may apply to the RMS or AI Labs Internship programs, but not both. However, all applicants may be considered for both programs, depending on qualifications and project needs.

Application Procedures

Complete the electronic application form. On the application form:

• Choose up to two research areas in which you are interested and provide written statements about your interest in the area(s) of research and how your experience aligns with the project.
• Attach a copy of your curriculum vitae (preferably as a PDF).
• If you are (or have been) actively enrolled in a graduate program, attach a copy of your graduate transcripts (unofficial copies are acceptable).
• Download the recommendation form and share it with your recommenders. The link to the recommendation form is on the application. Recommendations should come from an academic advisor, a professor who is familiar with your work as it relates to the project of interest or an individual with whom you have worked on a closely aligned project. ETS will only accept two recommendation forms.Recommendations should be sent electronically to internfellowships@ets.org and must be received by February 1, 2021. If you would like to download the recommendation form to send to your recommenders before submitting your application, you can save your application information for completion later.

Deadline

• The application deadline is February 1, 2021.

Decisions

• Applicants will be notified of selection decisions by March 31, 2021.

Duration

• Ten weeks: June 1, 2021–August 6, 2021

Compensation

• $7,500 salary

Eligibility

• Current full-time enrollment in a relevant doctoral program
• Completion of at least two years of coursework toward the doctorate prior to the program start date

Selection

The main criteria for selection will be scholarship and the match of applicant interests and experience with the research projects.

ETS affirmative action goals will be considered. We strongly encourage students from under represented groups and backgrounds to apply. Late or incomplete applications will not be considered.

For more information please visit:

www.ets.org/research/internship-fellowship/summer-rms-intership

The AI Research Labs’ work drives the innovation and development of teaching and learning technologies that are grounded in research and powered by next-generation AI. The Labs are dedicated to working closely with end-users to uncover real-world needs, and co-designing and prototyping solutions to meet those needs. Our staff is made up of research and learning scientists, software developers, research engineers, user experience researchers and designers, instructional designers and producers, and product owners.

Applying for an Internship in the AI Research Labs

Interns accepted into the AI Research Lab summer program will participate in user needs, discovery and exploration, solution ideation and validation, capability and prototype development, iterative user validation and data-driven solution optimization. We work in agile development teams to apply the best of foundational learning and cognitive science to the design, development and testing of solutions to meet educator and learner needs. Upon the completion of the program you’ll have the opportunity to present your findings to teams across R&D.

Applicants who have interest and expertise in the following would be a great fit for this program:

• learning or cognitive science
• software development
• AI and ML engineering
• user experience research and/or design
• instructional design
• product ownership

Note: Applicants may apply to the RMS or AI Labs Internship programs, but not both. However, all applicants may be considered for both programs, depending on qualifications and project needs.

Application Procedures

Complete the electronic application form. On the application form:

• Identify the Lab that you are interested in and provide a written statement about your interest in the area(s) of research in the Lab and how your education and experience align with the work of the Lab.
• Attach a copy of your curriculum vitae (preferably as a PDF).
• If you are (or have been) actively enrolled in a graduate program attach a copy of your graduate transcripts (unofficial copies are acceptable).
• If you were accepted into a graduate program and deferred enrollment, attach proof of acceptance.
• Download the recommendation form and share it with your recommenders. The link to the recommendation form is on the application. Recommendations should come from an academic advisor, a professor who is familiar with your work as it relates to the project of interest, or an individual who you have worked with on a closely aligned project. ETS will only accept two recommendation forms. Recommendations should be sent electronically to internfellowships@ets.org and must be received by February 1, 2021. If you would like to download the recommendation form for sending to your recommenders before submitting your application, the option to save your application information for later is available.

Deadline

• The application deadline is February 1, 2021.

Decisions

• Applicants will be notified of selection decisions by March 31, 2021.

Duration

• Ten weeks: June 1, 2021–August 6, 2021

Compensation

• $7,500 salary

Eligibility

• Completion of bachelor's degree
• Actively enrolled or accepted into a graduate program aligned to a Lab focus (students who have deferred enrollment due to extenuating circumstances will be considered)

Selection

The main criteria for selection will be the match of applicant interests and experience with the focus of the Labs.

ETS affirmative action goals will be considered. The Research Labs value building teams of individuals from diverse backgrounds and with diverse experiences. We strongly encourage students from underrepresented groups and backgrounds to apply. Late or incomplete applications will not be considered.

For more information please visit:

https://www.ets.org/research/internship-fellowship/summer-ai-research-labs-internship

Internship in Educational Measurement

National Board of Medical Examiners Internship

Summer 2021 Internships in Assessment Science and Psychometrics

June 7 - July 30, 2021   Philadelphia, PA

This year’s internship will be virtual, due to uncertainty surrounding the Covid-19 pandemic. 

Interns will interact with other graduate students and NBME staff and will present completed projects or work-in-progress to NBME staff. Internships typically result in conference presentations (e.g., NCME) and sometimes lead to publication or dissertation topics. 

Requirements 

• Active enrollment in doctoral program in measurement, statistics, cognitive science, medical education, or related field; completion of two or more years of graduate coursework.
• Experience or coursework in one or more of the following: test development, IRT, CTT, statistics, research design, and cognitive science. Advanced knowledge of topics such as equating, generalizability theory, or Bayesian methodology is helpful. Skill in writing and presenting research. Working knowledge of statistical software (e.g., Winsteps, BILOG; SPSS, SAS, or R).
• Interns will be assigned to one or more mentors but must be able to work independently.
• Must be authorized to work in the US for any employer. If selected, F-1 holders will need to apply for Curricular Practical Training authorization through their school’s international student office and have a social security number for payroll purposes. 

Compensation 

Total compensation for the two months is approximately $8000. 

Research Projects 

Interns will help define a research problem; review related studies; conduct data analyses (real and/or simulated data); and write a summary report suitable for presentation. Projects are summarized below. Applicants should identify 2 projects by number that they prefer to work on. 

1. Exploring Response Process Validity Evidence for a Medical Licensing Examination Program. Passing scores on the United States Medical Examination (USMLE) sequence are intended to signal readiness for the unsupervised practice of medicine in the United States. This project will examine response process validity evidence for USMLE score interpretations. USMLE is comprised of three computer-based multiple-choice question examinations: Step 1, Step 2 Clinical Knowledge, and Step 3. For each multiple-choice question on each Step examination, various testing features are available to examinees. These features include the ability to 1) highlight examination text, 2) cross out examination text 3) take notes in a blank window, and 4) view lab values. This project will examine the associations among performing these activities and performance on the examinations. Inferential statistical analyses will be done for all examinees as well as by examinee subgroup, defined by such characteristics as gender and location of medical school. Among other responsibilities, an intern for this project would help to design the study, manage and manipulate a large complex data set, conduct statistical analyses, and interpret and present/describe results. 

2. Virtual Performance Assessment. The United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills Examination (Step 2 CS), in existence since 2004, is undergoing a revitalization. Before the planned rollout of the restructured exam, the NBME will be conducting multiple research studies. Possible projects include work on generalizability, quality assurance metrics, or evaluation of scoring/equating designs for this virtual examination. 

3. What Determines Physician Competence? The USMLE assesses the competence level of future physicians based on the system of tests that check the breadth and depth of medical knowledge, as well as clinical and doctor-patient communication skills. In addition to the USMLE scores, a plethora of exam scores are collected throughout medical school and residency. The goal of this project is to examine the relationships between the various intermediate educational medical tests and the ultimate physician competence indicators. 

4. Interactive Psychometric Dashboards. Contemporary methods of communicating score information are evolving beyond fixed displays on traditional 8 ½ by 11-inch page layouts. Additionally, score users demand more flexibility and specificity to draw tailored inferences in support of evidence-based decisions. For this project the intern will contribute to the development of dashboard products using R Shiny and participate in engagement efforts with internal and external customers. 

5. Using NLP to Explore Item Bias and Test Validity. The use of Natural Language Processing (NLP) within applications, in conjunction with psychometric theory and statistics, has shown great promise for providing innovative information that was not previously available in the item and test development processes of high-stakes licensure programs. This project will focus on the use of basic NLP techniques and application development intended to analyze text in large item banks, to improve item and test development and provide more evidence for test fairness and the validity of test scores. An intern with little or no NLP experience, but who is interested in learning more about this field, may be suitable. Interns with previous NLP experience will have the possibility of incorporating more advanced NLP techniques and analyses into their work, depending on the project scope and other projects underway at NBME. 

6. Measurement Instrument Revision and Development. This project will continue ongoing work on revising a commonly used measurement instrument so that the appropriate inferences can be made about medical student well-being. Duties may include the following: working with subject-matter experts to revise the existing items; conducting think-alouds with medical students; developing a pilot measure of potential items; exploratory and confirmatory factor analysis of initial pilot results to gather structural validity evidence; developing a larger survey to gather concurrent and discriminate validity evidence with the revised measure; and administration and evaluation of the larger survey. 

7. Qualitative Analysis of Focus Group and Interview Data. The RENEW (Re-examining Exams: NBME Effort on Wellness) task force at NBME is focused on understanding the relationship between the pressures of high-stakes licensure examinations and medical student well-being. As part of this work, a series of focus groups and interviews have been completed, but this data has not been exhaustively utilized and analyzed. The intern assigned to this project will apply qualitative and mixed-method methodologies to learn more about the information gathered during the focus groups and interviews. 

8. Computer-Assisted Scoring of Constructed Response Test Items. Relying on subject matter experts to score constructed-response test items is expensive, time-consuming, and introduces natural scoring variations inherent when using human raters. Recently the NBME has developed a computer-assisted scoring program that utilizes natural language processing (NLP) to mitigate these issues. The two main components of the program are (1) ensuring that the information in the constructed response is correctly identified and represented; and (2) building a scoring model based on these concept representations. Current areas of research surrounding this project include (but are not limited to): refining quality control steps to be taken prior to an item being used in computer-assisted scoring; linking and equating computer-assisted scores with human rater scores; evaluating a scoring method based on using orthogonal arrays; and developing metrics that assess item quality and test reliability when computer-assisted scores and human scores are used to make classification decisions. The final project will be determined based on a combination of intern interest and project importance. 

9. Modern Test Construction Methods. The art and science of constructing an efficient and effective high-stakes exam requires balancing several competing aims. These theoretical and statistical considerations are explicitly enumerated when constructing an exam, resulting in a set of constraints that require complex algorithms to build secure test forms. Recent research has suggested that some of these constraints may be unnecessarily restrictive, resulting in test item banks that aren’t efficiently utilized and increased costs for test development and construction. An intern for this project would review current practices for exam construction and examine which constraints may be relaxed while still administering an exam with appropriate domain coverage and exam security considerations. 

Application 

Candidates may apply by going to https://nbme.applicantpro.com/jobs/. A cover letter outlining experience and listing project interests by number, along with a current resume, are required. Application deadline is January 31, 2021. 

All applicants will be notified of selection decisions by February 26, 2020. 

Overview 

NBME offers a versatile selection of high-quality assessments and educational services for students, professionals, educators, and institutions dedicated to the evolving needs of medical education and health care. To serve these communities, we collaborate with a comprehensive array of professionals including test developers, academic researchers, scoring experts, 

practicing physicians, medical educators, state medical board members, and public representatives. 

Together with the Federation of State Medical Boards, NBME develops and manages the United States Medical Licensing Examination. In addition, we are committed to meeting the needs of educators and learners globally with assessment products and expert services such as Subject Examinations, Customized Assessment Services, Self-Assessments, the International Foundations of Medicine , and Item-Writing Workshops. 

We also provide medical education funding and mentorship through the Latin America Grants, Stemmler Fund, and Strategic Educators Enhancement Fund, which serve to advance assessment at educators' and health professionals' own institutions. 

NBME employs approximately 30 doctoral level psychometricians and assessment scientists, as well as several MDs specializing in medical education. Staff is recognized internationally for its expertise in statistical analysis, psychometrics, and test development. 

Learn more about NBME at NBME.org. 

Diversity, Equity, and Inclusion: 

At NBME, we continue to innovate and improve how we fulfill the evolving needs of the health care community. This commitment starts and ends with the people at NBME. By recruiting and empowering talented individuals from various disciplines and backgrounds, which includes professionals with diverse life experiences, abilities, and perspectives, NBME can take a well-informed, robust approach to advancing medical education and assessment for years to come. Learn more about NBME at NBME.org.