Understanding and Improving the Reasoning Capabilities in Vision-Language Models

Vision-language models (VLMs) are considered augmentations of text-only large language models (LLMs) with additional access to the visual world. However, it is known that VLMs typically perform worse than text-only LLMs on reasoning tasks that involve only text (e.g., math word problems), raising the question of whether the benefits of visual augmentation justify the loss in reasoning capabilities. In this project, we will systematically benchmark and analyze the reasoning performance drop of VLMs and propose technologies to improve reasoning in VLMs through both performance-driven analysis and interpreting the internal hidden states of the models.

Faculty Supervisor:

Freda Shi

Student:

Partner:

Layer 6 AI

Discipline:

Computer science

Sector:

Finance and Insurance; Professional, scientific and technical services

University:

University of Waterloo

Program: