Undergraduates experiment with artificial intelligence, apps, and more (including slime!) to make technology more accessible

Where is the best place to put an interpreter on a TV screen? Can a filter allow signers to be anonymous — but still understood? What if an app offered medical consent forms in American Sign Language? These were just some of the research questions that students tackled at Gallaudet this summer during the Research Experiences for Undergraduates Accessible Information and Communication Technologies (AICT REU) program. Funded by the National Science Foundation (NSF), the 10-week course invites students to experiment with cutting-edge tools, and teaches them how to analyze and present their findings.

“Typically we have hearing and deaf students working together to solve a problem. Our goal is to increase the number of deaf researchers and allies — hearing students who get how things work,” says Dr. Raja Kushalnagar, director of the B.S. in Information Technology and M.S. in Accessible Human Centered Computing programs, who has run the AICT REU since 2014.

He and Dr. Christian Vogler, director of the Technology Access Program research group, selected 11 students (a mix from Gallaudet and other schools) to live and work on campus this summer. They matched the participants with projects based on their backgrounds and interests, and along with other Gallaudet faculty, they mentored the students and guided them through the research process.

This year’s group was especially engaged and collaborative. “All of the students had sign language skills for the first time,” Vogler notes. Bringing people with different perspectives together is critical in this field, he explains. “With accessibility, you cannot take a one-size- fits-all approach. You have to design for all, and at the same time design for one,” he says.

In pairs, the students conducted studies, which required recruiting participants, taking them through a set protocol, and collecting results. For example, Melanie Miga, a student at the University of Connecticut, and Gallaudet’s Richard Taulbee were assigned to figure out how deaf and hard of hearing people can best communicate with Amazon’s Alexa virtual assistant. Since people often interact with this technology while cooking, they purposely made a bit of a mess.

“I have an Alexa at home and I always use it with dirty hands,” explains Miga, who is hearing and can easily use voice commands. “We tried to simulate that experience,” Taulbee says. So they asked their subjects to follow a recipe for making slime, and then gave them the option to interact with Alexa through a screen-based app, or by standing in front of a video camera and either using ASL or gestures. Because the technology does not exist yet for the latter two options, Miga watched participants from another room and voiced the commands to Alexa.

This is called “The “Wizard of Oz” technique, and seeing the surprised reactions was entertaining. “One participant signed, ‘Lights on,’ and when it worked, signed, ‘Thank you, Alexa,’” Miga recalls. It was also enlightening to learn that no one liked the app — which required pausing to wash the slime off their hands — and there was a strong preference for ASL. “Users don’t want to memorize new commands,” Taulbee says.

Miga, Taulbee, and the other students in the AICT REU presented their projects in a poster session attended by Gallaudet faculty and staff, as well as representatives from the Federal Communications Commission and NSF. Provost Dr. Khadijat K. Rashid, ’90, remarked to attendees that this program is an extension of the university’s proud history of advocating and fighting for deaf people’s rights. Dr. Wendy Nilsen, NSF Deputy Division Director for the Division of Information and Intelligent Systems, told the students, “This is our future and we know it.”

Standing next to displays about their projects, students fielded questions about their work and the experience. Gallaudet students Sahej Walia and David Kanter, who studied the viability of a filter for ASL Anonymization, went over some of the issues their participants faced. “Two-handed signs were hard to understand. ‘Sandwich” looked like ‘prayer,’” explained Walia, who hopes that improved technology can address this problem. “I would love to see it go to the next level.”

The projects all offer further opportunities for research, both by adjusting experiment designs and taking advantage of new developments. Gallaudet students Kafayat Alabi and Minchan Kim explained how they created shortened versions of captions using ChatGPT, and want to monitor how more advanced artificial intelligence could improve their results. Finishing the project has given Alabi the confidence to continue this work. “No matter how hard it is, you can do it. You just have to push through,” she said.

The passion of the AICT REU group impressed Hannah Benjamin, a student at the New Jersey Institute of Technology. “We’re all here to make the world more accessible for everyone,” she said. “For undergrads, there are not a lot of opportunities like this to make things that will make a difference in the world.”