Define–Assess–Learn: Development of an AI Chatbot and Pilot Evaluation of Its Effects on Student Learning

This project will develop and pilot-test a dual-function Generative AI chatbot to support student learning in an undergraduate course. The system will integrate: (1) a definition-retrieval component that provides course-aligned explanations drawn from the textbook, lecture slides, and instructor-approved materials, and (2) an assessment component that evaluates students’ written explanations of key concepts by comparing them to official course definitions and offering targeted feedback on accuracy and completeness. Together, these functions are designed to help students clarify concepts, check their understanding, and identify gaps during self-directed study.

After development, a pilot study will be conducted to evaluate usability, feasibility, and initial learning effects. The pilot will inform a future large-scale experimental study. Students will be assigned to one of three groups: (a) a traditional classroom group with no AI support, (b) a definition-only AI group that can use the chatbot for concept lookup, and (c) a definition-plus-assessment AI group that can both retrieve definitions and assess understanding. Pre/post-tests, usage logs, and short surveys will be used to determine whether AI-supported learning improves conceptual mastery and whether the assessment function offers additional benefits beyond simple definition retrieval.

Faculty Supervisor:

Xingwei Yang

Student:

Partner:

Purdue University

Discipline:

Sociology

Sector:

Artificial Intelligence; Education; Technology

University:

Toronto Metropolitan University

Program: