Data Scientist (AI)

TL;DR

Data Scientist (AI/ML): Developing evaluation and benchmarking datasets for the Quick Suite AI platform with an accent on synthetic data generation and LLM-as-a-judge settings. Focus on creating high-quality ground truth datasets and identifying model failure patterns to ensure enterprise-grade AI safety.

Location: Hybrid; must be based near hirify.global offices in Seattle, Santa Clara, or New York

Salary: $136,000 – $212,800 annually

Company

hirify.global Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, serving everything from startups to Global 500 companies.

What you will do

• Design and develop comprehensive evaluation and benchmarking datasets for AI-powered features.
• Leverage LLMs for synthetic data corpora generation and quality assessment using LLM-as-a-judge settings.
• Create high-quality ground truth datasets with question-answer pairs across diverse domains.
• Lead human annotation initiatives and conduct model evaluation audits to ensure data relevance.
• Perform statistical analysis to measure model performance and identify failure patterns.
• Build scalable data pipelines and tools to support continuous benchmarking efforts.

Requirements

• 2+ years of professional data scientist experience.
• 3+ years of experience with SQL, Python, or other statistical software (R, SAS, Matlab).
• 3+ years of experience in machine learning/statistical modeling tools and techniques.
• 1+ years of experience working with or evaluating AI systems.
• Master's degree in a STEM field or equivalent professional experience.
• Must be based in or near Seattle, Santa Clara, or New York.

Nice to have

• Ph.D. in a STEM field.
• Experience defining benchmarks for assessing GenAI model performance.
• Previous experience in a ML or Data Scientist role at a large technology company.
• Proven ability to communicate complex concepts through written and verbal communication.

Culture & Benefits

• Flexible hybrid work model combining in-person collaboration with focused build time.
• Comprehensive health insurance including medical, dental, vision, and prescription coverage.
• Financial benefits including 401(k) matching, sign-on payments, and restricted stock units (RSUs).
• Paid time off and parental leave.
• Supportive culture with employee-led affinity groups and mentorship resources.