Quandt, Malzkuhn secure NSF grant for signing avatars research

The Office of Sponsored Programs (OSPRS) congratulates Dr. Lorna Quandt and Ms. Melissa Malzkuhn, principal investigators, on their award from the National Science Foundation in the amount of $849,999. This award is for a project titled New Dimensions of American Sign Language (ASL) Learning: Implementing and Testing Signing Avatars and Immersive Learning (SAIL 2).

The highly-spatial three-dimensional nature of American Sign Language (ASL) has created a serious barrier to technology-supported ASL instruction. What if ASL learners could access high-quality ASL instruction from native sign language instructors through a virtual reality-based, game-like environment? This project launches from prior work on the NSF-funded Signing Avatars and Immersive Learning (SAIL) project. The SAIL project yielded a working prototype of an immersive sign language learning environment in virtual reality. The current project expands past the prototype stage into a fully-fledged ASL learning experience. 

In the new version of SAIL, called SAIL 2, the research team is developing a more complete system where users enter virtual reality and interact with signing avatars (computer-animated virtual humans built from motion capture recordings) who teach users ASL vocabulary. Access to signed language is key to healthy development for many deaf individuals, but it remains a major challenge when access to high-quality ASL instruction is limited by time and resources.

SAIL 2 sets a foundation for greater access to learning ASL, which has the potential for improving the lives of deaf children and adults. The project focuses on developing and testing this entirely novel ASL learning tool and fostering the inclusion of underrepresented minorities in STEM. This work has the potential to substantially advance the fields of virtual reality, ASL instruction, and embodied learning.

Immersive virtual reality is particularly well suited for highly spatial signed languages. The SAIL 2 project leverages head-mounted virtual reality and high-quality signing avatars to create a gamified ASL-learning system. SAIL 2 will be the only ASL learning system in virtual reality that does not require the user to wear specialized gloves or other peripheral devices. The project develops a functioning version of the comprehensive SAIL 2 system, and user testing during the design process guides the details of the development. Key features of the system include sign recognition through hand tracking cameras, corrective feedback, and a gamified experience. Following the design and development of SAIL 2, the research team conducts behavioral research to evaluate the learning outcomes of SAIL 2. Evaluation of specific learning outcomes includes both the understanding of ASL vocabulary and the accuracy of sign production. Because of the embodied nature of signed language, mechanistic measures of the neural substrates of learning, including engagement of the sensorimotor cortices, are obtained through electroencephalography (EEG). The patterns of neural oscillatory activity provide insight into short-term changes in brain activity associated with using SAIL 2. The cognitive neuroscience experiment builds on previous research identifying the neural processes supporting sign language perception, and overall this project extends technological advances in high-fidelity motion capture recordings, avatar creation, and virtual reality.

Details of the award are as follows:

• Dr. Lorna Quandt and Ms. Melissa Malzkuhn Principal Investigators
• Ph.D, in Educational Neuroscience
• National Science Foundation
• Award No.: 2118742
• Dollar amount of federal funds awarded: $849,999 from 10/01/2021 to 09/30/2024
• Percentage total cost of the program financed with federal funding: 100%
• Percentage and dollar amount of the total costs funded by non-governmental sources: 0%

Congratulations to Dr. Quandt and Ms. Malzkuhn!