Research Agent Evaluation Harness and Methodology

Tiptree builds AI research assistants for ML scientists. These agents help users discover relevant prior work, validate claims, assemble evidence with citations, and produce high-trust research artifacts (e.g., literature reviews, benchmark comparisons, technical memos).
A core bottleneck in deploying research agents is evaluation: unlike standard software, agent behavior is nondeterministic, multi-step, and tool-dependent, and failures often look superficially plausible (e.g., missing critical papers, hallucinated citations, or silent omissions). This makes it hard for both researchers and product teams to trust outputs and to iterate quickly on system design.
This project will develop a rigorous evaluation harness and methodology for research agents, grounded in best practices from agent evaluation (e.g., task/trial/graded-outcome framing, transcript analysis, pass@k and pass^k reliability metrics) and validated against scientific research agent benchmarks (e.g., AstaBench and MLGym). The work will also incorporate behavioral evaluation methods (e.g., Bloom-style targeted behaviors) to measure properties that matter in real research settings: citation faithfulness, source quality, appropriate tool selection, stopping criteria, and interaction quality.
Anticipated benefits for Tiptree include: (1) a reusable evaluation suite that functions as a regression test bed for product iteration, (2) higher-trust research agent performance for customers in academia and R&D, and (3) publishable methodology and results that strengthen Tiptree’s technical credibility and commercialization pipeline.

Faculty Supervisor:

Bang Liu

Student:

Partner:

Tiptree Advanced Systems Corporation

Discipline:

Computer science

Sector:

Information and cultural industries; Professional, scientific and technical services

University:

Université de Montréal

Program: