How a business-university partnership is building smarter virtual cockpits for pilot training

Pilot training was once a very tactile experience, with pilots receiving feedback from buttons, switches, and knobs that would give them a physical indication that an action had been taken.

Full-flight simulators are multi-million-dollar investments that provide highly realistic, hands-on training experiences. To extend and reinforce this training, the aviation industry has introduced immersive AR and VR cockpit environments that allow pilots to practice procedures outside the simulator. These technologies offer additional training opportunities with minimal operational costs while complementing, rather than replacing, the physical interaction and tactile feedback provided by full-flight simulators.

“Recognizing these interactions accurately and in real time is difficult because cockpit controls can involve complex movements, and even small delays or recognition errors can negatively affect the realism and effectiveness of the training experience,” explained Joshua Castillo, a Machine Learning Researcher at CAE, a global leader in simulation-based training for aviation and defence. Joshua is working to help the company bridge the industry gap.

Teaching a virtual cockpit to read your hands

While Joshua was a computer science graduate student at Concordia University, he joined CAE through a Mitacs internship, working on emerging technologies to enhance pilot training through AR and VR systems.

One of his research directions was gesture recognition for virtual cockpit interactions. “It addresses a key challenge in immersive training: enabling pilots to interact naturally with cockpit controls using only their hands,” said Joshua. “This capability has the potential to support a new category of immersive training experiences, complementing traditional simulation platforms in providing access to high-quality training.”

With this technology, a student pilot sits inside a virtual cockpit wearing an AR or VR headset. Instead of physically interacting with controls or handheld devices, users perform natural cockpit actions that the system translates into virtual control inputs, including knob adjustments, switch activations, and more advanced yoke and side-stick interactions

“Because the user cannot physically feel the control, the technology must recognize the intended action based solely on movement and provide feedback through the simulation. The goal is to make these interactions feel as natural and realistic as possible while supporting effective pilot training,” Joshua said.

Under the supervision of Dr. Marta Kersten-Oertel, former Associate Professor at Concordia, now at the University of Ottawa, CAE’s research has focused on understanding how virtual systems can better interpret and support natural interactions within a simulated cockpit setting.

“The objective has been to evaluate different technological avenues and better understand the opportunities and limitations associated with advanced human-computer interaction in AR and VR training applications,” he explained. Solutions that can provide a realistic, responsive, and reliable training experience suitable for complex operational environments are of particular importance.

Mitacs connects CAE with the next generation of talent

By helping connect industry with academic institutions, Mitacs has enabled CAE to engage with leading academic expertise, contribute to the development of highly qualified talent, and explore research opportunities across a variety of research and development domains.

Through the collaboration with Concordia University and Mitacs, CAE’s work contributes to the development of specialized talent in Quebec and Canada, strengthens links between academia and the aerospace simulation sector, and supports innovation in immersive training technologies. It also helps Canada maintain its position in aerospace and simulation by building capabilities and expertise that can support future training solutions.

For Joshua, it’s also been meaningful on a personal level. “Beyond supporting the research itself, Mitacs provided me with valuable experience working on real-world challenges,” he said.

The research partnership continues and includes exploring practical applications of emerging technologies “In the context of this project, improvements to immersive pilot training could help pilots develop and maintain critical skills more effectively, ultimately contributing to aviation safety and benefitting society as a whole,” said Joshua.

“Mitacs has created a true win-win partnership between industry and academia. It allows organizations like CAE to explore innovative research opportunities while giving students like me access to real-world challenges and industry expertise. By bringing these communities together, Mitacs helps accelerate innovation, develop talent, and ensure that academic research can create meaningful impact beyond the university setting.”

Joshua Castillo, Machine Learning Researcher, CAE

About Mitacs

Mitacs delivers the return on innovation Canada needs now by connecting big ideas, people, and ambition to turn potential into results. Mitacs co-funds innovation partnerships that get Canada’s businesses and researchers where they need to go.

Mitacs partners innovate faster. Commercialize sooner. Compete globally. Scale and grow stronger. We open the door to opportunity, creating jobs and equipping students and postdoctoral researchers with skills, industry experience, and professional networks.

A not-for-profit organization founded in 1999, Mitacs is funded by the Government of Canada, the Government of Alberta, the Government of British Columbia, Research Manitoba, the Government of New Brunswick, the Government of Newfoundland and Labrador, the Government of Nova Scotia, the Government of Ontario, Innovation PEI, the Government of Quebec, the Government of Saskatchewan, and the Government of Yukon.