Towards understanding people’s interactions with a video question-answering AI system

In recent years, there has been a rise in the development of interactive AI systems and applications where the user follows a procedure to complete a task, including (instructional) fitness and cooking apps to name a few examples. Most of these efforts focus around video question-answering (QA), where a system is built to answer questions a user asks when watching an instructional video. However, little is understood of the usability of these interactive AI technologies and what types of questions a user might ask in a realistic scenario, when they are following the procedure, in an instructional video.
In this project we aim to observe the user in real time, where they are actually asked to follow a procedure demonstrated in a video, and interact with it via asking questions. Our goal is to understand the nature of the questions that users would ask, and whether they would find it beneficial to be able to interact with such an AI system via natural language queries.

Faculty Supervisor:

Khai N. Truong

Student:

Partner:

Samsung Electronics Canada

Discipline:

Computer science

Sector:

Manufacturing

University:

University of Toronto

Program: