System level defenses for Enterprise AI Agents

(1) the main activities of the partner
ServiceNow develops a platform for client organizations to manage and automate large-scale processes across various industries. In 2020, ServiceNow acquired Element AI to strengthen its presence in the Artificial Intelligence (AI) research landscape and the Canadian AI ecosystem. This acquisition enabled the development of AI-driven products that improve the platform’s capabilities. ServiceNow Research has made significant contributions to the field of foundation models, notably in Natural Language Processing (NLP), and has a strong presence in developing generative models for different data domains.
(2) the challenges the partner aims to solve through this project
The project seeks to address the security and safety challenges posed by autonomous AI agents in enterprise settings. Specifically, it aims to mitigate risks such as data exfiltration, prompt injections, privilege escalation by developing system-level defenses that make AI agents secure against adversarial attacks. The goal is to maintain the utility and autonomy of AI agents while safeguarding against security threats from AI agents operating in untrusted environments.
(3) the anticipated social or economic benefits of the project for the partner organization(s)
The project is expected to enhance the security of autonomous AI agents, which will safeguard sensitive data and protect against malicious attacks. This will not only boost productivity and efficiency in enterprise settings but also foster a safer digital environment. Ultimately, the opensource release of the findings of this project will contribute to the broader community by setting new standards for AI security and promoting trust in AI technologies.

Faculty Supervisor:

Irina Rish

Student:

Partner:

ServiceNow Canada

Discipline:

Computer science

Sector:

Professional, scientific and technical services

University:

Université de Montréal

Program: