Voice-to-Text & Text-to-3D Content Pipeline in Virtual Reality using Unity

This research project aims to create a system that allows users to communicate with a virtual environment using their voice. The system will convert the user’s voice commands into text and use natural language processing techniques to understand the user’s intent. Then, the system will generate 3D content based on the identified intent using AI models. Finally, the system will integrate with the Unity engine, allowing users to manipulate the 3D content in a virtual reality environment using their voice. The expected outcomes of this project are a pipeline that enables users to interact with virtual environments using their voice, as well as an understanding of the human-AI integration in the context of virtual reality. The project will also provide valuable experience in the VR/AR/MR Metaverse.

Faculty Supervisor:

Michael Barnett-Cowan

Student:

Partner:

National Technical University of Ukraine

Discipline:

Computer science

Sector:

Health and Related Sciences & Technology; Artificial Intelligence; Entertainment and Media

University:

University of Waterloo

Program: