Research Recommendations and Implementation

Research Recommendations and Implementation
In 2010-2012, the ASLPI evaluation system underwent validity and reliability studies by an external research and consulting firm. The lead researcher traveled to Gallaudet University to present about the findings to the Gallaudet faculty and administration (outcomes posted above).

Upon conclusion of this two year research study, the following recommendations were prioritized by ASL Diagnostic and Evaluation Services:

• Enhance training and refresher trainings with a greater emphasis on rating accuracy, consistency, and strict performance appraisals for attaining and maintaining evaluator status.
• Develop more comprehensive Functional Descriptions for 0-5 proficiency levels, including Functional Descriptions for the plus (+) levels.
• Establish stricter standards for agreement among raters by transitioning from an adjacent rating approach to absolute agreement for rating decisions. The spring 2013 semester was an intensive focus to enhance the training and refresher training programs. Additional components and activities were developed with specific focus on interviewing strategies and techniques to elicit functional language at each proficiency level, as well as focus on rating accuracy
• During the summer of 2013 from June through August, The ASLPI conducted a comprehensive and careful analysis of the Functional Descriptions for the 0-5 proficiency levels, including the plus (+) levels. After 500+ hours and critical analysis of 90+ video recorded ASLPI evaluations, updated and enhanced Functional Descriptions were finalized.

At the beginning of fall 2013, all ASLPI Evaluators were required to participate in a mandatory training. Adherence to stricter standards for rating agreement among evaluators was also instituted. The absolute agreement approach for rating decisions was implemented as is recommended for high stakes testing.

Following the training and implementation of the stricter rating expectation (absolute agreement for proficiency level decisions), The ASLPI established a monitoring system. Prior to distribution, every rating decision went through a review process to examine inter- and intra-rater reliability. This ensured application and effectiveness of the training. At the end of fall 2013, The ASLPI conducted a rating reliability study across all evaluators. The reliability study included the 275 evaluations conducted. Rating reliability for the entire pool of ASLPI Evaluators was determined to be 84%. For high stakes language proficiency testing, reliability must be 80% or above.

The ASLPI is continuing to closely monitor all evaluators to ensure that a shared mental model is consistent and evaluators are applying the same operational definitions of the proficiency constructs.