Building Spatio-temporal Transformers for Egocentric 3D Pose Estimation

Using a fisheye head-mounted camera to estimate human pose in 3D has become increasingly popular in recent years due to its ability to capture activities in unconstrained environments. Egocentric 3D human pose estimation (HPE) has a number of challenges due to self-occlusions and strong distortions. Intermediate heatmap-based representations have been found to be effective in reducing distortion, however self-occlusion remains a challenge, and is the proposed focus of this internship. The goal of the project is to build on previous work, the Ego-STAN project, to improve the performance of that method in the context of 3D HPE, particularly to make it more robust to occlusion. The broader goal is to have a method suitable for cutting-edge motion tracking applications such as activity recognition, surgical training, and immersive XR applications.

Faculty Supervisor:

Paul Fieguth

Student:

Partner:

National University of Kharkiv

Discipline:

Computer science

Sector:

Other; Artificial Intelligence

University:

University of Waterloo

Program: