Study of LLM Bias on French dialects

As Large Language Models (LLMs) evolve, they have become increasingly adept at avoiding overt expressions of bias. However, more subtle forms of prejudice, particularly those related to dialect, can still persist. This project aims to investigate latent biases in LLMs by analyzing how they respond to prompts phrased in different French dialects.
We will compare LLMs responses across several French dialects, including Québec French and standard French from France. By systematically varying the dialectal features of each prompt and analyzing the model’s tone, content, and inferred stereotypes, we aim to quantify and characterize French dialect–based variation in model behavior.

The goals of this project are threefold:
-Develop domain expertise in evaluating sociolinguistic bias in LLMs, particularly as it pertains to underrepresented French dialects in Canada.
-Build benchmark datasets and protocols that can be used to assess and track dialect bias in current and future generations of language models.
-Publish research findings to contribute to the broader conversation on fairness and inclusivity in AI.

Faculty Supervisor:

Jackie Cheung;Samantha Jackson;Samantha Jackson

Student:

Partner:

Armilla AI

Discipline:

Computer science

Sector:

Finance and Insurance

University:

McGill University; University of Toronto

Program: