Optimizing Lightstage Capture for High Fidelity 3D Facial Reconstruction

Ubisoft is one of the world’s largest video game studios, specializing in 3D open-world games that require precise 3D character representations. In particular, achieving high-quality facial features is crucial, as humans are highly sensitive to small details in facial expressions. Currently, creating 3D
facial representations first requires a Lightstage capture pipeline. This process begins with taking highly detailed photographs of actors’ faces, which are then processed through a Multi-View Stereo (MVS) [1] algorithm to generate an extremely dense and irregular 3D mesh. The mesh is then registered and simplified into a regular mesh ready for use in game development. However, this pipeline is costly, requires significant manual labor, extensive memory storage, high computational power, and relies on third-party software. For instance, a 3D sequence of 20 minutes currently requires 150TB of storage, and 8 weeks of processing time with eleven working PCs. This project aims to optimize the MVS pipeline, helping Ubisoft save precious time as well as financial,
material (working PCs, GPUs, storage) and human resources (artists and performers). Specifically, the project will focus on creating improved metrics to measure MVS reconstruction quality of facial features and leverage those metrics to innovate in the field of 3D facial reconstruction.

Faculty Supervisor:

David Lindell

Student:

Partner:

Ubisoft Toronto

Discipline:

Computer science

Sector:

Information and cultural industries

University:

University of Toronto

Program: