Development and evaluation of principled distance measures for high-dimensional random representations such as LLM and process-based behavioral embeddings, to support interpretable, and fair inference in scoring, test security, and feedback generation

Distance Measures for High-dimensional Representations

This project addresses a foundational challenge in the use of high-dimensional representations, such as large language model (LLM) embeddings and process-based behavioural embeddings: how to define and evaluate distance measures that are principled, interpretable, and suitable for scientific inference. Current practice often treats embeddings as fixed vectors with ad-hoc geometric assumptions. This project instead reconceptualises embeddings as stochastic representations and develops distance measures grounded in statistical, psychometric, and information-theoretic principles.

The partner organisation, Educational Testing Service (ETS), develops and administers educational assessments used worldwide. Many assessment-related systems increasingly rely on embedding-based models to represent language, responses, and behaviours. ETS is committed to ensuring that decisions derived from these models—such as clustering, scoring, and anomaly detection—are valid, reliable, and fair. Project activities focus on designing principled distance metrics, evaluating their properties, and demonstrating their relevance to assessment use cases.

Anticipated benefits for ETS include improved interpretability and validity of embedding-based models, stronger fairness and robustness arguments, and reduced operational risk. More broadly, the project advances trustworthy AI methodologies that support fairer and more reliable use of AI in education.

Faculty Supervisor:

Feng Ji

Student:

Partner:

ETS Canada

Discipline:

Sociology

Sector:

Education; Professional, scientific and technical services

University:

University of Toronto

Program: