Assessing the Risks of Self-Reinforcing Attacks on Generative AI Systems

(1) the main activities of the partner
ServiceNow develops a platform for client organizations to manage and automate large-scale processes across various industries. In 2020, ServiceNow acquired Element AI to strengthen its presence in the Artificial Intelligence (AI) research landscape and the Canadian AI ecosystem. This acquisition enabled the development of AI-driven products that improve the platform’s capabilities. ServiceNow Research has made significant contributions to the field of foundation models, notably in Natural Language Processing (NLP), and has a strong presence in developing generative models for different data domains.
(2) the challenges the partner aims to solve through this project
The project addresses security risks in commercial AI software systems, focusing on a self-reinforcing attack strategy that progressively poisons databases, influencing AI behavior. It aims to explore these risks and develop defense mechanisms against such stealthy and subtle attacks. This includes examining how untrusted sources, such as web content or emails, can inject malicious data into storage systems, posing significant threats to AI systems using retrieval-augmented generation.
(3) the anticipated social or economic benefits of the project for the partner organization(s)
The project aims to enhance the security and reliability of AI systems by understanding the threats associated with self-reinforcing attacks and developing defenses to protect against them. By open sourcing the findings of this work, we will contribute to safer and more secure AI integration in enterprise environments. This will not only protect sensitive data and reduce potential economic losses but also help bolster user trust of AI technologies and increase the rate at which these technologies can be safely adopted.

Faculty Supervisor:

Graham Taylor

Student:

Partner:

ServiceNow Canada

Discipline:

Computer science

Sector:

Professional, scientific and technical services

University:

University of Guelph

Program: