Technology
Expert Talent Pool, Evals and Benchmarks (Associate)
Clera
Ashby
- Location
- remote, Not specified
- Type
- FullTime
- Salary
- Salary not specified
- Posted
- 16 Nov 2026
ABOUT THE ROLE Join a team that designs and runs human-led evaluations of AI models. You will help measure model capabilities by creating rigorous questions, rubrics, and comparative assessments, contributing to better understanding of AI performance. WHAT YOU'LL DO - Write questions and tasks that test specific model capabilities. - Develop rubrics that define strong, partial, and weak answers. - Create reference answers and edge-case scenarios to challenge models. - Grade model outputs against rubrics and explain each score clearly. - Analyze incorrect answers, group error patterns, and document findings. - Compare multiple models using the same prompts. - Validate benchmarks and flag questions that may no longer provide reliable measurements. - Support small studies of how people evaluate or interact with model outputs. WHAT WE'RE LOOKING FOR - Zero to five years of professional experience; prior AI-data work is not required. - Experience or training in QA, test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or a subject area involving rigorous question writing. - Experience writing test cases, exam questions, rubrics,
ABOUT THE ROLE Join a team that designs and runs human-led evaluations of AI models. You will help measure model capabilities by creating rigorous questions, rubrics, and comparative assessments, contributing to better understanding of AI performance. WHAT YOU'LL DO - Write questions and tasks that test specific model capabilities. - Develop rubrics that define strong, partial, and weak answers. - Create reference answers and edge-case scenarios to challenge models. - Grade model outputs against rubrics and explain each score clearly. - Analyze incorrect answers, group error patterns, and document findings. - Compare multiple models using the same prompts. - Validate benchmarks and flag questions that may no longer provide reliable measurements. - Support small studies of how people evaluate or interact with model outputs. WHAT WE'RE LOOKING FOR - Zero to five years of professional experience; prior AI-data work is not required. - Experience or training in QA, test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or a subject area involving rigorous question writing. - Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria. - Ability to use spreadsheets or Python to organize and analyze results, and to identify and document error causes. - Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo, model comparisons, or user and rating studies is useful. - Clear, professional written communication. COMPENSATION & BENEFITS Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked. LOCATION Fully remote. Projects typically run from one week to several months.