Research Engineer – Benchmarking

Mercor is seeking a Research Engineer to own benchmarking pipelines, evaluation systems, and failure analysis workflows that directly inform how we train and improve frontier language models. You will define how we measure tool use, agentic behavior, and real-world reasoning, working at the intersection of engineering and applied AI research. This role requires strong applied research background, coding skills, and comfort with ML models and evaluation code.

• Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning
• Build and operate LLM evaluation systems end-to-end, including scoring, dashboards, and reporting
• Run systematic failure analysis on model outputs and categorize failure modes
• Create and refine rubrics, automated evaluators, and scoring frameworks
• Quantify data usability, quality, and impact on key benchmarks
• Collaborate with AI researchers and applied AI teams to align evals with training objectives