AI-based sign language interpreter - ON-541

Project type: Research
Desired discipline(s): Engineering - computer / electrical, Engineering, Computer science, Mathematical Sciences
Company: Deaf AI
Project Length: 6 months to 1 year
Preferred start date: As soon as possible.
Language requirement: English
Location(s): Toronto, ON, Canada; Canada
No. of positions: Up to three (3)
Desired education level: Postdoctoral fellow
Search across Mitacs’ international networks - check this box if you’d also like to receive profiles of researchers based outside of Canada: 
Yes

About the company: 

Deaf AI improves the experience of using the digitalized world for the deaf community. Develops technology accessibility to a higher level for deaf people, thereby making technology more inclusive. Create a better future of communication by making a transition from human-based service to AI-based service in sign language interpretation. Deaf AI is an example of AI4Scoail Good to specify how AI serves human beings especially people with hard of hearing and hearing loss. Deaf AI makes sign language interpreting more affordable, accessible, and available to service users and service providers.

Describe the project.: 

The purpose of this project is to make a better and more effective communication tool for deaf people and to help them get more easily involved in the digitalized world by developing virtual avatars based on AI representing sign language interpreters.

The candidate will work on:

  • R&D
  • Traning the model
  • Developing the AI-based sign language interpreter by ML and other python libraries like MediPipe

At first, the model will be developed then will be trained on datasets. The inputs for the model are voice (voice-to-text) or text, and body recognition of sign language interpreters. For illustration, let’s imagine there is a video that a person is speaking and on the corner of the screen there is a sign language interpreter. Develop a program to recognize the hand gestures of a sign language interpreter with some libraries like MediPipe (this is the first input), then voice or text recognition (subtitle) should be targeted simultaneously (the second input). Both inputs should be used to train the model concurrently. In this video, for example, for the first 10 seconds, the voice and hand motions are detected, the model should be trained on the first 10 seconds of the voice and the first 10 seconds of hand motions together at once.
In fact, the model will receive two inputs at the same time, the first is video voice and the second is motion detection of sign language by the interpreter.

We have additional information to share on this with prospective candidates.

Required expertise/skills: 

AI expertis particularly in computer vision, deap learning, and machine learning.