Supporting post-WIMP interactions in Mixed Reality with Spatial Computing and Edge Computing

Mixed Reality (MR) allows the display of virtual content, for instance with holograms, that can merge to variable degrees with the physical environment of a user by taking into account the context and topology of the user’s environment. By its mobile, spatial and context-aware nature, MR allows to interact with virtual content with new interaction paradigms that are closer to the natural interactions we have with everyday objects. In this project, we will work on two interrelated objectives: (i) to support the online recognition of physical objects in the user’s environment via the sensors of mixed reality headsets, and (ii) to design interactions for MR based on real-life objects, where these can be used as effectors to interact with virtual content. To achieve these goals, our team will use object recognition (e.g., Convolutional Neural Network (CNN)) and action recognition algorithms (e.g., Recurrent Neural Network (RNN)). To overcome the limited computational resources on MR headsets, the distribution of these algorithms will be evaluated on edge to cloud computing devices. As leaders in virtualisation and in edge to cloud computing support solutions,

Faculty Supervisor:

Charles Gouin-Vallerand;Kévin Bouchard

Student:

Partner:

VMware

Discipline:

Computer science

Sector:

Professional, scientific and technical services

University:

Université de Sherbrooke

Program: