TL;DR

Sr Applied Scientist, AI Agent Evaluation (AI): Designs and implements evaluation frameworks for AI agents, including benchmarking tools and annotation systems, ensuring consistent performance across diverse use cases. Focus on developing expertise in AI agent systems and devising new research solutions for customer problems.

Location: Must be based in the US

Salary: $150,400 - $260,000/year

Company

%hirify_global% Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform.

What you will do

Design and implement evaluation frameworks for AI agents.
Develop expertise in AI agent systems and evaluation.
Identify and devise new research solutions for customer or business problems.
Lead the design, implementation, and delivery of solutions for scientifically-complex problems.
Write clear narratives and documentation describing inventions, solutions, and design choices.
Actively mentor and develop other scientists and engineers.

Requirements

3+ years of building machine learning models for business application experience.
PhD, or Master's degree and 6+ years of applied research experience.
Experience programming in Java, C++, Python or related language.
Experience with neural deep learning methods and machine learning.
Experience using managed ML/AI solutions.
Deep expertise in at least one AI/ML discipline such as NLP, reinforcement learning, etc.

Nice to have

Experience with modeling tools such as R, scikit-learn, Spark MLLib, MxNet, Tensorflow, numpy, scipy etc.
Experience with large scale distributed systems such as Hadoop, Spark etc.
7+ years of experience applying scientific methods to solve complex AI problems at scale.
Experience mentoring junior scientists and engineers.
Advanced knowledge of large language models, including fine-tuning, evaluation, and responsible AI practices.

Culture & Benefits

%hirify_global% values diverse experiences and encourages candidates to apply even if they do not meet all preferred qualifications.
We value work-life harmony and strive for flexibility as part of our working culture.
Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences.
Endless knowledge-sharing, mentorship and other career-advancing resources are available to help you develop into a better-rounded professional.