MTS – Evals in Zürich

You’ll develop benchmarks for multimodal models and agents, working with domain experts to define useful tasks and assess the results.

This can involve evaluating an answer about a video or reviewing the steps an agent took in an application .

You’ll study whether our tests measure the intended skills and help the team understand what the results mean for further research.

• What you’ll work on
• Define tasks, success criteria, and evaluation datasets with researchers and domain experts.
• Implement checks for generated files, application state, and model responses, and compare automated scores with expert assessments.
• Run model comparisons, analyze failures, and identify opportunities for reward hacking.

Strengthen tasks and automated checks to prevent models from earning high scores without completing the intended work.

Read the traces.

• Investigate data leakage, ambiguous tasks, and sources of variation in scores; maintain versioned tasks and reproducible evaluation runs.
• What you’ll bring
• Strong Python skills for processing datasets, implementing evaluation logic, and analyzing results with tools such as Num Py and pandas.
• An understanding of statistics and experimental design, including sampling, uncertainty, and confounding factors.
• Experience evaluating models or testing complex software, and an interest in investigating unexpected results.
• Clear technical writing that explains evaluation methods, findings, and limitations.
• You take initiative and follow through on your work. You’re principled in your decisions, open about uncertainty, and collaborative in how you solve problems.
• Nice to have
• Experience with model APIs, Hugging Face Datasets, pytest, or tools for visualizing experimental results.
• Work on benchmark design, reward hacking, data contamination, or human evaluation.
• Familiarity with CAD tools and their scripting APIs, video analysis, or computer-use agents.
• A Ph D in a relevant field and research publications are preferred, but neither is required.
• Comprehensive health, dental, and vision coverage.
• Visa sponsorship and relocation support.
• A professional development budget, the best AI tools, GPUs, a Mac, and the subscriptions you need.
• Team off-sites and conference attendance.
• Support with transportation costs.
• Gym membership and sports club subscriptions to support your wellbeing.
• A food stipend with nearly full meal coverage.
• #J-18808-Ljbffr