An intelligent framework for scalable sign language synthesis

Over 5% people across the world are hard of hearing, and require assistance in navigating public transportation/airports. This project attempts at automated sign language synthesis through a virtual avatar for improving day-to-day life of people with hearing impairment. The industry partner has developed a preliminary framework to conduct animation on virtual avatars to play back sign language. However, the current data pipeline to create the animations is cumbersome, error prone, and not scalable. The proposed project will create an intelligent framework for accurate and scalable sign language animation on virtual avatars. The framework will address common sign language synthesis issues such as self-occlusion, motion blur by employing novel deep neural network-based approaches. A large dataset will be collected to train the model, utilizing state-of-the-art head-mounted displays (e.g. Apple Vision Pro) instead of expensive motion capture set up, so that sign language experts from across the world can contribute. The proposed project is expected to create core technology for the industry partner that can not only help them expand their operations across other cities and countries, but also benefit Canadians as a whole by making public transportations and airports more accessible.

Faculty Supervisor:

Naimul Khan

Student:

Partner:

Deaf AI

Discipline:

Engineering

Sector:

Professional, scientific and technical services

University:

Toronto Metropolitan University

Program: