Perception-Guided Planning for Long-Horizon Sequential Manipulation

The project explores how to enable robots to perform multi-step tasks in realistic, changing environments by integrating visual information in different aspects of planning. Instead of assuming a perfect internal map of all objects, we will use camera input both to understand the scene and to guide the robot’s decisions during planning and execution. Demonstrations and state-of-the-art vision models will be used to suggest useful intermediate goals and to notify the planner when something has gone wrong, like dropping an object during grasping. In the end, the goal is a robot that can plan its actions, monitor its progress visually, and recover from mistakes more reliably, which is important for practical use in homes, warehouses, and other everyday settings.

Faculty Supervisor:

Igor Gilitschenski

Student:

Partner:

Technische Universität Berlin

Discipline:

Computer science

Sector:

Artificial Intelligence; Information and Communications Technology (ICT)

University:

University of Toronto

Program: