Zero-to-Hero: Data Augmentation with LLMs and Human Feedback

Intent classification is a highly popular task within research communities and industries. Having a system that can classify intents of emails or messages from users can lead to a wide range of applications such as ticket routing and issue resolution. However, training such models require a large set of labeled data that might not be readily available. In this project work, we aim to present a system where only a few examples are required to be given for each intent. At each cycle we train an intent classifier and use a large language model (LLM) to generate extra examples using the examples in the dataset. Those generated examples are carefully selected as humans are involved to verify them. Concretely, the LLM generates a set of examples that are ranked based on scores given by the models involved in the system. To have more diversity and variations in the generated examples, we allow the LLM to generate new examples using previously generated examples. This system has the potential to train a powerful intent classification method with extremely low human effort that could significantly speed up the process of addressing customer requests.

Faculty Supervisor:

Olga Vechtomova

Student:

Partner:

ServiceNow Canada

Discipline:

Computer science

Sector:

Artificial Intelligence; Technology; Information and Communications Technology

University:

University of Waterloo

Program: