A/B Test Simulation for Free-to-play Mobile Games - ON-545Project type: Research
Desired discipline(s): Computer science, Mathematical Sciences, Mathematics, Statistics / Actuarial sciences
Company: Uken (aka Uken Games)
Project Length: 6 months to 1 year
Preferred start date: As soon as possible.
Language requirement: English
Location(s): Toronto, ON, Canada; Canada
No. of positions: 1
Desired education level: Master's
About the company:
Uken builds games that people everywhere love. Who Wants To Be A Millionaire? and Jeopardy! World Tour are the top 2 trivia games, engaging millions of global players every month. In 2020, Uken released Ava’s Manor, a solitaire game where users renovate and decorate the manor while discovering mysterious clues and dabbling in a romantic love story. Uken’s foray into narrative games is an exciting opportunity for the company.
We consider our Ukenites to be data-driven designers, technologists, and artists. Through collaboration, we pair the craft of design with feedback and data to deliver a pace and rhythm in each game session that immerses our audience and engages them for years. We have a quirky, nerdy culture that fosters creativity, collaboration, quality ideas, and a data-driven mindset. We believe in moving quickly and improving constantly and that mantra is reflected in the weekly updates we make to our games and internal technologies. We have an appetite for pushing technical boundaries but with a focus on practical use.
Describe the project.:
Currently, to evaluate the impact of a change to game balance, a statistical A/B test is run against a live user population. Through analysis of this user data, a decision between the new, (proposed) feature versus the existing (control) feature is made and applied to all users. This process is resource-demanding in terms of development time, time is taken to run the test, and potential negative impact on the user population.
This project focuses on building a solution to simulate game changes against a model of how users will behave under a different set of conditions. The goal of the simulator is to run a “virtual A/B test”, to estimate the impact of a potential change without having to go through the full A/B testing process. A comprehensive data set of past user actions in-game, which is updated daily, is available as an input into this simulation tool to drive accuracy in modelling user behaviour.
Some of the unique problems in creating a solution in the mobile free-to-play gaming space are the high flexibility of in-game systems, and how frequently they change. In order to test features changes with this solution, system parameters should be accessible as inputs as often as possible. Due to the high frequency of game feature releases, the underlying game systems are also subject to frequent change. The design of the model or simulation tool should be robust to maintain accuracy when new systems are added or system parameters are significantly changed - even if that data is based on past user actions, where the potential new feature does not exist.
The system should be built using Python, and be able to be updated and run with relative ease by data analysts within the company.
Due to the wide scope of modelling and simulation required for this project, we are looking for individuals with expertise in one or more of these fields:
- User behaviour modelling
- Economic system modelling
- Building simulation systems
- Statistical A/B testing and analysis
- AI development