Data Scientist (AWS Generative AI)

TL;DR

Data Scientist II (AWS Quick Data): Engineering high-quality evaluation and benchmarking datasets for generative AI features in an enterprise AI platform with an accent on LLM-as-a-judge, synthetic data generation, and human annotations. Focus on designing ground truth datasets, conducting statistical analysis of model performance, and developing safety/fairness evaluations to ensure enterprise-ready AI.

USA: WA Seattle, CA Santa Clara, NY New York. Hybrid work model near one of our U.S. hirify.global offices.

USA WA Seattle: $136,000–$184,000; CA Santa Clara: $157,300–$212,800; NY New York: $153,400–$207,500 annually

Company

AWS Quick Suite is an enterprise AI platform combining generative AI search, research, agents, automations, and business intelligence for tens of thousands of users.

What you will do

• Design and develop evaluation/benchmarking datasets for AI-powered features
• Leverage LLMs for synthetic data generation and quality assessment using LLM-as-a-judge
• Create ground truth datasets with question-answer pairs across domains
• Lead human annotation initiatives and model evaluation audits
• Conduct statistical analysis to identify failure patterns and guide improvements
• Collaborate with ML scientists/engineers to translate insights into product improvements
• Build scalable data pipelines for continuous evaluation
• Contribute to Responsible AI safety and fairness datasets

Requirements

• 2+ years data scientist experience
• 3+ years SQL, Python (or R/SAS/Matlab), ML/statistical modeling
• 1+ years working with/evaluating AI systems
• 1+ years creating mathematical textbooks/research papers/educational content
• Master's in STEM or equivalent experience
• Experience applying theoretical models in applied environments

Nice to have

• Ph.D. in STEM
• ML concepts for reasoning/problem-solving
• Experience in ML/data scientist role at large tech company
• Defining/creating GenAI benchmarks
• Multi-team cross-disciplinary projects
• Quantitative analysis for business decisions
• Communicating complex concepts

Culture & Benefits

• Inclusive team culture with affinity groups, CORE, AmazeCon events
• Mentorship, career growth, knowledge-sharing resources
• Work/life harmony with flexibility
• Hybrid options balancing office collaboration and focus time
• Comprehensive benefits: health insurance, 401(k) matching, PTO, parental leave