The Certainty of the Standards in Uncertain Times

By Megan Welsh posted 12-13-2020 19:52


Vince Verges, Florida Department of Education

Uncertain Times Then

Many years ago in my former life as a Naval Flight Officer, I learned how valuable standard procedures can be when things get chaotic. In 1992, I was a navigator aboard a four-person tactical Navy jet that experienced problems over the Olympic National forest in a very remote part of Washington state. We ended up ejecting over that forest, and while everyone survived, I suffered the loss of my left hand during the ejection. I ended up dangling from a tree for a short while before somehow getting myself down, and then had to tend to my wounds for the next few hours before a Navy search and rescue helicopter extracted me from the dense woods and delivered me to Bremerton Naval Hospital. After I recovered, I was eventually medically retired and then moved into the field of education and educational measurement, where I have been amazed at how transferable so much of my military experience has been, hard-earned though it was. I’ll illustrate my point for you below.

Uncertain Times Now

In the current state of affairs that we find ourselves in with respect to the ongoing COVID-19 pandemic, I am once again re-visiting the importance of grounding my work to the greatest extent possible in the stability offered by standardized guidance and procedures.  Just as we had little control over many of the circumstances we found ourselves in that day in the clear, blue skies over Washington state, so now is our field faced with numerous threats to the validity and reliability of test results that we’ll be collecting and analyzing in the coming years. What we’ve studied and been trained to do in the field of educational measurement are critical foundations to successfully navigating the challenges we’ve already seen and will continue to face.

Trust the Guidance Borne of Experience and Expertise

In the Navy, we relied on standardized procedures outlined in volumes of manuals, most running to nearly 1,000 pages including appendices, with each volume gaining chapters over the years based on lessons learned from incidents and accidents.  While not nearly as massive, and certainly not a life-or-death document, the field of educational measurement offers us a similar critical guide in the Standards for Educational and Psychological Testing (Standards), which most of you know is the joint product of the American Educational Research Association (AERA), the American Psychological Association (APA), and NCME. Published collaboratively by the three organizations since 1966, it represents the gold standard in guidance on testing in the United States and worldwide.

As events unfolded for me in the moments leading up to and especially during the time I ejected, and until I was rescued, the outstanding training I received in flight school allowed me to manage almost by rote the aspects of the situation that I did have control over. There are standard procedures for preparing to eject, ejecting, descending in a parachute, preparing to land, self-administered first aid, survival, signaling, and preparing for rescue. While there was no standard procedure for how to get out of a tree using only one hand, there was enough other training to help bridge some of those gaps. Likewise, there is clearly no road map in the Standards for the specific circumstances that test owners and test users find themselves in now, but there is ample guidance that will help inform the work, and will help us make the best decisions that we can.

What Do the Standards Have to Say About Test Administration?

One of the important messages that the Standards gives us is with respect to variation:

Although variations may be made with the intent of maintaining score comparability, the extent to which that is possible cannot often be determined. Comparability of scores may be compromised and the test may not measure the same construct for all test takers.

Other such key messages we find in the Standards should be viewed in an entirely new light, and are important to attend to.

Flexibility in administration, timing, presentation, or supports may be appropriate so that test takers with disabilities may have access to the test content, but these now also must be considered with respect to our new environment. However, the Standards also tell us that:

Care should be taken so that variations do not fundamentally alter the construct being measured. Judgments regarding the suitability of such adjustments must be made by qualified individuals, and must be consistent with guidelines set forth by test publishers and users.

I urge readers to pay specific attention to what the Standards tell us about test administration procedures as we look at ways to address assessment in entirely new learning environments. Broad topics covered in this area in the Standards include changes or disruptions to test administration procedures; testing environment; appropriate instructions, practice, and support; and maintaining test security and integrity. As we look at innovative ways to administer tests in challenging conditions, it becomes no less important to attend to these factors when developing solutions.

Specific Recommendations from the Standards

  • Ensure test administrators are provided all necessary training resources and test administration guidance.
  • Schedule ample time for training on test administration and security.
  • Allow time to recruit proctors for what may be unique administration situations.
  • Ensure computer-based test readiness, including preparations for release of new software/browsers/updates.
  • Provide adequate access to sample/practice test materials so examinees are familiar with the computer-based tests they would be taking.
  • If modifying any assessment instrument, test users and test takers must be provided with adequate guidance on test length, number of test days, and session timing, as applicable.
  • If administering a new type of test to measure the impacts of COVID-19, treat any such test the same as any other test administration to ensure that all examinees have the same fair testing experience.
  • Any relaxation of test security policies or requirements would send the wrong message regarding the seriousness with which we take this or any future test administrations.

In Closing

In an ideal situation, the Standards tell us that testing conditions should be consistent with those expected, and prevalent when norms and interpretive data were gathered. That is indeed a very tall order for many testing programs in this new world we find ourselves operating in. Of no small consequence is how that guidance is viewed in light of the burgeoning field of remote-proctored, at-home test administrations. However, if we keep this and other guidance in mind as we build our solutions to test administration, we always have a foundation against which we can check our assumptions about the validity and reliability of the results collected from such assessments, and it guides us in what we need to caution test users about when considering the uses of those results.