Detection of anomalous emotional responses using attention mechanisms for deep machine learning

Computer-based multimodal affect recognition methods fuse multiple informational channels, typically video, audio, and text, to resolve the emotional state of a monitored individual.
The proposed research aims to develop multimodal deep learning models to recognize anomalous emotional responses, which correspond to a deviation from the expected affective reaction for a particular context. Since multimodal affect recognition applications involve large inputs such as video and audio frames and text passages, training deep neural networks to recognize anomaly presents significant challenges as the model may be unable to optimally maintain the spatial and sequential information. Hence, we propose to employ attention mechanisms to increase the emphasis on relevant spatial and temporal relationships in the input data. Attention mechanisms have been mainly applied for natural language processing applications; however, we hypothesize that we can develop analogous approaches for video and audio signals to improve model performance.

Faculty Supervisor:

Hussein Al-Osman

Student:

Partner:

Université Grenoble Alpes

Discipline:

Computer science

Sector:

Education

University:

University of Ottawa

Program: