The Program Committee in 2021 welcomes proposals to conduct 15-minute demonstrations of an innovation – something creative, innovative, or novel that you would like to share with our membership. You may want to demonstrate an RShiny Package or a thought experiment you use in your teaching. Perhaps you’ve developed an instrument that classroom teachers need to see or a macro that researchers need to see. If you’ve discovered or created a resource that would benefit the measurement community but that doesn’t quite fit the research manuscript orthodoxy, please consider submitting a demonstration proposal. There is no paper required, and you may propose either an individual 15-minute presentation or a 60-minute group of four related demonstrations.
We intend for the demonstrations to provide a forum where new ideas can be shared and stress-tested. We want participants to walk away with an idea, an app, or some other solution they can use. As such, proposed innovations must be free for participants, no strings attached. Proposals that aim to sell commercial products at the conference will be rejected. We will, however, welcome proposals that introduce free innovations that run on commercial software (e.g., SAS macros).
To encourage participation, friendly competition, and a bit of fun, the Program Committee (with the assistance of volunteer judges) will recognize outstanding achievement in the demonstration of a measurement innovation. Award categories will include (1) the William of Ockham Award recognizing an elegant solution to a vexing problem; (2) the Rube Goldberg Award for an innovation that summons uncommon complexity to solve a simple problem; and (3) best in show for demonstration by a graduate student. The demonstration proposal review criteria, which these award categories reflect, are provided on pp. 7-8.
Specific Guidelines for Demonstrations
Proposals for demonstration presentations must identify all contributors in the proposal. Proposals must consist of (a) an abstract of no more than 50 words for individual, 100 words for group (for inclusion in the final program), (b) a summary of the demonstration in no more than 500 words for individual, 1000 words for group, (c) any software packages required (if applicable), and (d) references, tables, and figures as appropriate. The Program Committee will reject proposals that exceed these word limits. The summary should accomplish two things:
- Introduce the innovation itself. Describe the problem it addresses, the typical users (e.g., classroom teachers, researchers), and, if available, evidence of the innovation being put to use. One way to do this quickly and clearly is through a value proposition statement (“This helps X do Y by doing Z”). Regardless of the structure proposers adopt, the summary should clarify the practical utility and implications of the innovation and should not be written as a business case, a product roadmap, or marketing collateral. Moreover, proposers should not assume that their innovation must rely on or have anything to do with technology. We are interested in the modified eyeglasses that let you observe the counterfactual even in low-light conditions, but we are also interested in the paradox you have identified and/or resolved or the song you wrote to teach the assumptions of ANOVA. Advances in the science of measurement have depended at least as much on creativity, logic, and argumentation as they have on technology; we see no need for that to change in 2022.
- Describe the format of the demonstration (e.g., lecture, brief hands-on training). The Program Committee is interested in novel, interactive presentation formats, but proposers’ first order of business should be describing a demonstration that is well-aligned with the innovation itself and that is feasible given time and technology constraints. For example, it is reasonable to expect that some innovations are best introduced through a lecture and guided tour rather than a hands-on activity requiring nonstandard technology (any technology needs above and beyond the equipment that is standard at conventional paper sessions must be supplied by the presenters).
The Program Committee expects that the demonstration format will be distinguished by its immediate payoff, among other things. Participants should walk away with a concrete new skill, insight, or technological support that they can leverage in their work without much additional research or training. Therefore, the Program Committee strongly recommends that proposers use the summary to briefly explain what participants will be able to do after the demonstration that they likely could not have done before it. Finally, it is bears repeating that we want to showcase some innovations where technology is instrumental and some where it is incidental. Both varieties will receive due consideration during proposal review.
Individual paper presentations
For those that are new to the Annual Conference, the individual presentation format is the most common submission format for a researcher or research team seeking to share their work with other NCME members. Proposals should describe a single paper written by one or more authors. The first author should be the primary presenter, although authors may present together. Individual presentations at the Annual Conference in Spring of 2022 will be in one of three formats: (i) a traditional lecture-style presentation of approximately 12-15 minutes, to be presented in a multiple-paper session with related papers grouped by topic by the Program Committee; (ii) Research Blitz session where the authors will have 5 minutes to summarize the critical aspects of their research, and will remain in the session for group discussions with session attendees, or (iii) a poster-style presentation using an electronic board in a 60- or 90-minute session. Authors must indicate their preference for (i), (ii), or (iii), although the Program Committee may override these preferences to resolve scheduling constraints in the final program. Authors will be notified of presentation format as part of the proposal notification communication.
Specific Guidelines for Individual Presentations
Proposals for individual presentations must be prepared for blind review – author names should not be included in the document. Proposals must consist of (a) a title of no more than 12 words, (b) an abstract of no more than 50 words (for inclusion in the final program), (c) a summary of research of no more than 800 words, and (d) references, tables, and figures as appropriate. The Program Committee will reject individual presentations proposals whose titles, abstracts, or summaries exceed the word limits or are not blind. References, tables, and figures do not count toward the word limits. The summary should include research questions, methods, and findings. The Program Committee also strongly recommends that authors include the practical implications of their research (see the Evaluation and Review Criteria below).
Coordinated paper sessions
The program committee defines a coordinated paper session as a set of papers organized around a central theme or topic. The session will be lecture-style presentation with 3 – 5 papers and a discussant. The session proposal should meet the following guidelines.
Specific Guidelines for Coordinated Paper Sessions
Proposals for coordinated paper sessions must identify all contributors – up to ten authors’ and presenters’ names should be included, not blinded. Proposals must consist of (a) a title for the session of no more than 12 words, (b) an abstract of no more than 200 words (for inclusion in the final program), (c) a summary of the coordinated paper session (in addition to the abstract) of no more than 1600 words, and (d) references, tables, and figures as appropriate. After entering this information, the next page will require you to submit specific descriptions for each participate in the coordinate paper session. We will reject proposals whose titles, abstracts, or summaries exceed the word limit (references, tables, and figures do not count toward the word limit). Organizers may use the 1600 words however they wish, for example, a conventional 4-paper symposium may have a 400-word introduction with four 300-word paper descriptions. Proposals should also identify a discussant/moderator where appropriate.
The Program Committee defines an organized discussion as a planned conversation among researchers and/or practitioners around a theme or topic. We encourage debates, panel discussions, and other innovative formats, especially those involving interactivity with the audience. Preferences will be given to proposal that are in line with the conference theme. The session proposals should clearly describe the format of the proposed session and meet the following guidelines:
Specific Guidelines for Organized Discussions
Proposals for organized discussions must identify all contributors – up to five presenters’ names should be included, not blinded. Proposals must consist of (a) a title for the session of no more than 12 words, (b) an abstract of no more than 200 words (for inclusion in the final program), and (c) a summary of the organized session (in addition to the abstract) of no more than 800 words that described the theme or topic of the discussion, the proposed format for the discussion, the significance or implications of the issues for discussion (including key questions that would be addressed), and the perspective (s) that each presenter would represent. The Program Committee will reject proposals who titles, abstracts, or summaries exceed the word limits. The proposed format should be clearly motivated and clearly described. Proposals should also identify a discussant/moderator where appropriate.
Two important notes on organized discussions at the 2022 Annual Conference
- Given the relative scarcity of empirical data over the past year, organized discussions were the only way some projects could share their work in 2021. In 2022 we intend to shift the focus somewhat to the science of measurement, which implies a stronger emphasis on empirical research. The Program Committee therefore expects to schedule significantly fewer organized discussions in 2022 than were held in 2021.
- Over the years, NCME’s debates have been very popular and very valuable events – introducing our members to both the burning questions and the leading voices in our field. NCME also values a diversity of voices in these events, and in 2022 we hope to schedule panels featuring speakers who’ve not yet had the opportunity to take part in or lead NCME’s organized debates.
We invite proposals for training sessions addressing themes relevant to the mission of NCME. In 2022 we are particularly interested in novel training sessions focused on equity-oriented concepts such as fairness, anti-racism and social justice, and approaches for taking up such concepts in the context of both classroom and large-scale assessment. These sessions are scheduled to occur prior to the conference. Presenters must indicate their preference for length (four hours or full day) and preference for the mode of instruction (virtual vs. in-person). Please provide the target audience, three keywords, and if this training session has been offered previously.
Specific Guidelines for Training Sessions
Proposals for training sessions must include the name(s) of the presenter(s) and consist of (a) a title for the session of no more than 12 words, (B) an abstract of no more than 200 words (for inclusion in the final program), (c) a summary of no more than 500 words, and (d) a draft schedule of proposed activities and topics to be covered during the proposed session timeline, no more than 500 words (this is excluded from the 500 word limit in the summary). The title should be as descriptive as possible to give NCME members a clear sense of what will be covered. The abstract should provide an overview of the session content, method of instruction, learning objectives, the intended audience, and laptop/software use. The summary should highlight the goals of the session, expertise of the presenter(s), and the relevance/importance of the topic. The schedule of proposed activities and topics should focus on what presenter(s) and participants will be doing during the session. Note: presenters are responsible for bringing any handouts needed for the session or providing participants with information about how to obtain any suggested texts.
Graduate Student Issues Committee Research Session
All graduate students are invited to submit a proposal for the Graduate student research session at the NCME Annual Conference. All presenters will follow the individual presentation guidelines above. All presenters in the 2022 NCME graduate student research session will use the electronic board format. Graduate students submitting their work for consideration in this session may submit either completed research OR research-in-progress.
Diversity Issues in Testing
The NCME Committee on Diversity Issues in Testing (CODIT) announces an opportunity for all organizers of and participants in coordinated sessions and organized discussions to nominate a session to be the NCME Diversity Issues in Testing Invited Session. We encourage any coordinated session or organized discussion that addresses or reflects diversity issues in testing, broadly conceived, to nominate their session by selecting the "I would like this session to be considered for the NCME Diversity Issues in Testing Invited Session" option during the regular submission process. All such proposals will be reviewed and selected under standard procedures regardless of nomination status. However, nominated proposals will also be reviewed by the CODIT for distinction as the NCME Diversity Issues in Testing Invited Session, and, if selected, will be given a prominent placement in the program. The NCME CODIT and the NCME Board hope that this encourages session proposals to incorporate and consider issues of diversity as they develop and present their research topics and findings.