August 2000 Board of Directors Meeting Minutes
Announcements and Review of the Agenda
President Fremer called the meeting to order, reviewed the agenda, and asked if there were any modifications or additions to the agenda.
Approval of the Minutes
Minutes of the April 24 and April 27, 2000 Board of Directors Meetings were approved with a notation that two names were misspelled in the document.
Dates of Next Meeting
The Board confirmed the dates previously selected for the next Board of Directors meeting in Chicago on December 2-3, 2000.
Test Standards
At the April meeting, the Board adopted a motion that implied NCME that the participation of NCME in the next Test Standards revision project should require that individuals who work at testing organization not be excluded from consideration to serve on the revision committee. Dr. Kolen reported on his conversation with APA about the issue. He thought it was unlikely APA could make a decision on the item at this point.
Dr. Russell reported that AERA considered the proposed management committee agreement at its June meeting. AERA approved the draft management agreement, which is silent on the question of the composition of the revision committee. To avoid the possible appearance of a conflict of interest, AERA also adopted a motion that is in opposition to allowing individuals who work at a testing organization from serving on the test standards revision committee. The arguments and rationale that NCME advanced for their position were presented to AERA. AERA's counter argument was that the extensive review and consultation inherent in the development of the Standards would ensure the Committee access to the requisite knowledge and experience.
Concern was expressed that AERA did not consult with NCME prior to its action. It was noted that the AERA position was in direct conflict with NCME policy. It was also noted that a compromise position of having on the revision committee members who had previously worked at a testing organization, but were not doing so currently, was not approved. However, it was not explicitly excluded.
Subsequent discussion suggested that NCME may not be disadvantaged by the new management agreement. There will be three co-chairs and between 12-16 members on the revision committee. It was further noted that the draft agreement stated "Presidents of the three sponsoring organizations shall each appoint four members to the Joint Committee." Presumably, NCME would have major influence over the appointment of one of the three chairs. It was suggested, therefore, that efficiency may be an advantage over representation. It was urged that NCME remain vigilant when the Management Committee and the Presidents of the three sponsoring organizations appoint the next Test Standards Revision Committee. Accordingly, a motion was passed that NCME approve the draft management agreement for the Standards for Educational and Psychological Testing.
The conversation then briefly turned to dissemination of the Standards. Concern was expressed that APA did not appear to be fully cooperative, as evidenced by the fact that no information about the availability of the Standards in their book store at APA's last Annual Meeting. The question posed about was what else NCME could do to make the Standards visible? It was suggested that NCME's Standards and Test Use Committee might be asked to think about the question of dissemination and possibly write an article about the Standards for EM:IP. Perhaps a "cliff notes" version of the Standards should also be prepared. Another consideration was that the Standards could be the subject of an ITEMS model. It was noted that JCTP submitted a proposal to develop a Casebook based on the new Standards. Discussion of dissemination strategies for the Standards with the Association of Test Publishers was another idea advanced.
The notion of having a Web version of the Standards was not supported, given the need to sell the Standards to fund the development of future revisions. It was suggested that NCME and AERA members have the opportunity to purchase the Standards at the Annual Meeting.
If was noted that an annotated guide to all the professional standards that related to educational measurement would be a useful reference document. There may be about 20 or so organizations that have developed standards that would be appropriate for inclusion in the document. The Standards and Issues Committee would be asked to explore the idea.
Consideration of the Standards concluded with the notion that an evaluation and commentary on of the Test Standards be organized as a Web activity. Such an on going data collection activity would be immensely valuable in guiding the next revision effort.
Joint Committee for Testing Practices
The Board reviewed a proposal from JCTP to develop a Casebook on the revised Standards for Educational and Psychological Testing.Each member organization of JCTP must review proposals for new JCTP activities. Therefore, NCME could stop the project at the outset. It was noted that NCME would have an opportunity to shape its development. Board members felt the idea for a Casebook was good. The following questions arose: Would NCME have an opportunity to endorse the final product? Who was the target audience? Was there a need for the Casebook? Who would be the publisher? What was the role of the consultants in the project? Would the document be easily accessible? It was felt that the sample case study provided in the proposal was not good and that other examples listed could be problematic. That is, the examples seem to go beyond the Standards and therefore would not reflect an interpretation of the Standards. A stated audience was thought to be too broad (e.g., teachers) and thus not realistic. It was further suggested that the intended audience be part of the formal review process. The Board indicated that it would welcome the opportunity to review a revised proposal that provided better or more appropriate examples of the case studies, reflected a more focused audience, and devised a plan that reflected the onerous work that would be required. President Fremer did indicate that he would draft a summary of the Board's concerns and suggestion, share it with the Board, and then transmit it to JCTP.
NCME Committees
Last fall, the Board suggested that for the sake of balance and function, the Graduate Student Issues Committee and the International Student Issues Committee be removed from the Membership and Recognition area and added to the Standards area. The Board also suggested that the Standards area be renamed Standards and Issues. In addition, it was suggested that Educational Testing Legislation and Policy Analysis be transferred from Outreach to Standards and that the Classroom Assessment Award be transferred from Membership and Recognition to Outreach. After discussion of the recommendations, the Board approved all the suggested changes except that Educational Testing Legislation and Policy Analysis Committee would be moved to the Administrative area. The changes will be effective in April 2001.
Membership Decline
Board members reviewed the data on the incremental decline in membership over the past several years. It showed a nearly 20% decline since 1995. It was recalled that the NCME membership promotion to Division 5 of APA the past year was totally unsuccessful. Various promotional ideas were considered. The most promising ones, based on history, were a mailing to divisions D and H of AERA and the mail lists from various conferences that are interested in educational measurement issues. The suggestions will be passed on the membership committee. It was requested that welcome letters, or e-mail acknowledgments, be sent to new NCME members.
Nominations Committee Report
It was suggested that there may be a perception that the nominations process for NCME office may not be open. Therefore, it was suggested that there be an open call in EM:IP or the Newsletter, at the breakfast meeting, and on the Website. The call should also explain the nominations and election process.
Oral History Project
Four years ago, Irv Lehman was asked to develop a history of NCME. Irv was unable to secure any of the information that he requested from former officers of the Council. The Board suggested that he conduct interviews with NCME members who were instrumental in the development of NCME. It was reported that Irv had expressed a renewed interest in undertaking the task. Dr. Fremer was requested to express the Board's continued interest in the project and their appreciation for his willingness to undertake the work. It was suggested that Irv prepare a memo detailing the type of questions or information he would ask of former officers, editors, and others. Past presidents, for example, might be asked to write a one page statement about NCME during the year of their presidency. Some Board members offered to provide graduate students to conduct some interviews with NCME members if Irv, and a committee he might form, identify the individuals and provide an interview script or protocol. It was noted that the Executive Board authorized $500 to assist Dr. Lehman for such expenses as travel to the Annual Meeting to interview members.
Annual Meeting
Dr. Fremer reported that the Program Committee welcomed suggestions for invited sessions for the Seattle Annual Meeting. It was suggested that a session that focused on recruitment of professionals and students into the measurement community might be highlighted. The off-site visit to the Microsoft Home of the future did not seem feasible given the travel time that would be involved.
OCR Guidelines
The U.S. Education Department's Office of Civil Rights had recently released a draft report on The Use of Tests When Making High-Stakes Decisions for Students: A Resource Guide for Education and Policy Makers. The final document was to be released in early fall. The question raised by the Board was that if NCME wanted to issue a press release when the document was released? What would NCME's position be on the issue? The statement should also be published in the Newsletter. Thus, NCME should get ahead start in preparing the statement. Dr. Russell would send Dr. Lane information on the deadline. The statement should express portions of the document that NCME supported and note other aspects that are the most troubling.
Recruitment Into the Field
Dr. Kolen reminded Board members that he convened a meeting at the last Annual Meeting to generate ideas for increasing the recruitment of students into the Measurement field. He noted that there were about 20 people in attendance. He distributed a report summarizing the discussion around four general topics: issues contributing to the recruitment problem; proposed solutions; drawing prospective students to the major; and next steps. He thought the brainstorming of ideas was beneficial. Among the suggestions advanced was the notion of creating terminal master's level programs with a special concentration in measurement. It was noted, however, that careful consideration should be given to the question if there would be jobs for someone with the skills that could be developed in a one or two year program. Many of the suggestions that were relevant to NCME would be disseminated to the appropriate NCME committees and programs. For example, several suggestions involved links and information that should be provided on the NCME Website.
On a related matter, Dr. Lane distributed a memo about her efforts to determine whether NCME would be able to identify potential candidates for educational measurement and statistics programs via the GRE Search Service. Based on intended area of study, self-reported GPA and GPA within major, it appeared that this is possible. Dr. Lane wanted to know if NCME was interested in using the service? The question was what, if any, cost would be incurred for such a service and the size of the list. For example, 10% of the list would produce about 22,000 names. Dr. Lane will have answers to the questions for the December Board meeting. It was felt the best time for a mailing to students would be in the early spring.
Publications
Dr. Frisbie reported that Del Harnisch, Website Editor, had agreed to continue his term through December 2001. The search for a new editor should begin the first of the year with an appointment to be made by the summer Board meeting in 2001. The Call for nominations for a new editor would ask interested candidates their vision for the Webpage and what changes would be contemplated.
Dr. Frisbie announced that he anticipated that the Publications Committee would have a slate of three candidates for Dr. Fremer's consideration as the next ITEMS editor by the time of the December Board meeting.
The Board approved the recommendation that prior ITEMS modules be included on the Website. The list of the modules should be listed on the Website to provide greater visibility. The question was raised that if ITEMS were published on the Website and not in EM:IP, would they count in academic considerations? They would, however, likely reach a broader audience than the NCME membership. The Board reaffirmed the policy that ITEMS modules be peer reviewed.
The Publications Committee will consider issues associated with the notion of publishing older issues of NCME journals (JEM and EM:IP) on the Website.
The Board approved the request of the new editor of JEM for a budget of up to $15,480 to assist with editorial work associated with the editorship. The Board also approved Dr. Dodd's recommendation of Ralph DeAyala as Review Editor for the journal.
The Board approved a request to increase the page allocation from six to eight pages for the next two issues of the Newsletter.
Dr. Frisbie highlighted a continuing concern that the number of submission to JEM was insufficient to sustain quarterly publication. The need to plan at least one or two special issues of the journal was reinforced.
Outreach
Dr. Lane suggested that the Outreach Committee needs structure and a better sense of direction or purpose for their work. It was suggested that the Committee look at the NCME Long Range Strategic Plan. Items 8A and 9 suggest that the Committee might survey local and state assessment directors to identify testing practices and policies NCME can inform. NCME could also build stronger relationships with district and state assessment directors though NCME's publications, annual meetings, and other collaborative initiatives. It was further suggested that the committee focus on a specific activity before expanding its purview. Developing relationships or partnerships with other organizations was seen as a priority. That would require an active monitoring and liaison with selected organizations. It also implied that the Committee needed a well-defined purpose and set of objectives to guide its work. Board members agreed to send Dr. Lane a list of organizations that they would recommend the Committee initiate a conversation for possible collaboration as well as thoughts on what type of relationship would be appropriate for such a partnership.
The Sunday morning session began with a blatant act of defiance of established norms. A few Board members did not sit at the same place as they were the previous day. An alliance was quickly and secretly formed among traditional thinking Board members. They vowed to vote the renegade members off the Board of Directors at the December tribal meeting.
Psychometric Software
Dr. Kolen distributed the final report from the Ad Hoc Committee on Software Issues in Educational Measurement. The Committee was charged to recommend steps that could be taken by NCME to facilitate the availability of information about specialized software used in educational measurement and the dissemination of such software. The report provided nine conclusions and recommendations for the Board's consideration. The Board recommended that a summary of the report be prepared for publication for the NCME Newsletter.
The Board was not ready to authorize a publication giving guidelines for software documentation. It was felt that further review was necessary and reactions to the notion should be solicited from the Association of Test Publishers. The idea of establishing a software exchange for the purpose of dissemination might be useful. The term "exchange" was a misnomer and it should be explicit that the proposed Software Information would not provide reviews of software.
The Publications Committee was asked to review the recommendations concerning dissemination of information, including documentation, about software used in educational measurement. The Standards and Test Use Committee will be asked to consider the desirability of NCME adopting a position on levels of software dissemination for users. It was noted with some curiosity that other associations apparently have not established standards or guidelines on this issue. Dr. Kolen agreed to work with the Committee chair, Brad Hanson, to flesh out the details associated with NCME establishing a position on software dissemination to facilitate practice and research in educational measurement.
Discussion concluded with a request that the President thank the Committee for their outstanding work.
Other
Dr. Fremer introduced for discussion the implications for the field of the increasing anti testing movement in the country. There appears to be some inconsistencies in the public's eye. For example, people like tests but perceive tests to be unfair. It was noted that many high-stakes decisions are being made on limited information. The critical question is, should NCME be doing anything to explain what is a proper role for testing in education? Media do not call NCME when writing an article that raises questions or issues about testing. NCME should be seen as a resource. That will only occur if NCME takes an activistic approach to the media. That would require NCME to develop a bank of articulate experts who would be available to talk with reporters. Dr. Fremer offered to draft a letter to the media for the Board's review. It would describe the organization generally and illustrate a few specific areas of expertise.
The AERA's statement on High-Stakes Testing was discussed briefly. The question was raised: Why was NCME not involved in the development of the statement? Dr. Russell provided background on the development of the document by AERA. It was requested that AERA be asked to indicate that NCME, a co-sponsor of the Standards for Educational and Psychological Testing, supported the principles and intent of AERA's position statement on High-Stakes Testing in PreK-12 Education. (Subsequent to the meeting, AERA approved the request and NCME's support is noted on the printed brochure.)
Dr. Fremer announced that he hoped to draft, for the Board's consideration, the 10 or so items that the general population need to know about educational measurement. He envisioned the document educating the public and parents about the nature, benefit, and purposes of sound educational testing programs.
Adjournment
President Fremer thanked Board members for their thoughtful participation and the meeting was adjourned.
Respectfully submitted,
William J. Russell
Executive Director