Low-cost LLM inference at the Edge

Seamlessly serving Large Language Model (LLM ) inference requests/queries generated from user devices such as smartphones, laptops, tablets, etc., is critical for LLM service providers. The current approach of serving these queries using large LLMs in the cloud is expensive and incurs high latency and energy costs. More importantly, it cannot meet the growing demand for LLM-powered applications. To address this problem, we propose using edge computing infrastructure to serve the LLM queries at the first hop on edge servers using cost-effective small language models (SLMs). To mitigate the low-quality response provided by the SLMs and to maximize the utilization of the compute and memory capacity of the edge server, we propose the use of an LLM router and a scheduler, and outline the main ideas for designing these modules such that the final LLM inference response by the system is both accurate and fast. The system and the ideas developed in this project will alleviate the scalability issue of LLM inference serving systems and contribute to the ubiquitous use of LLMs.

Faculty Supervisor:

Jaya Prakash Varma Champati

Student:

Partner:

Manipal University Jaipur

Discipline:

Computer science

Sector:

Artificial Intelligence

University:

University of Victoria

Program: