Data Scientist (AI)

TL;DR

Data Scientist (AI): Defining success metrics, designing experiments, and turning agent trace data into actionable insights for hirify.global's AI agent, the core of their product strategy. Focus on statistical rigor, connecting agent data to user outcomes, and influencing roadmap decisions for AI agent effectiveness.

Location: Hybrid in Foster City, CA (in-office Monday, Wednesday, and Friday)

Salary: $140K – $280K

Company

hirify.global is a platform that enables anyone to build applications using natural language, democratizing software development through AI-powered collaboration.

What you will do

• Design and analyze experiments to measure AI agent improvements with statistical rigor and practical tradeoffs.
• Define success metrics that connect agent trace data to user outcomes like successful deploys, retention, and revenue.
• Build the semantic layer for agent data in partnership with data engineering to enable self-serve analysis.
• Surface insights from trace analysis to identify failure modes, successful patterns, and opportunities to improve agent effectiveness.
• Partner with AI engineering, product, and leadership to translate data into roadmap decisions and critical agent strategy discussions.
• Create dashboards and reporting to surface agent performance metrics for the AI team and executives.

Requirements

• 5+ years of experience in data science, analytics, or a quantitative role with a focus on product, growth, or experimentation.
• Deep experimentation expertise including A/B testing, experiment design, power analysis, handling skewed data, and interpreting results.
• Strong SQL skills and experience designing data models for high-volume event data, with experience in dbt or similar tools.
• Proficiency in Python and data science libraries such as pandas, scipy, and statsmodels.
• Ability to translate ambiguous questions into structured analysis and communicate findings clearly to technical and non-technical stakeholders.
• Bias toward action, shipping insights that influence decisions, not just dashboards.

Nice to have

• Experience with LLM or AI agent evaluation, understanding prompt-response patterns, or model quality measurement.
• Background in high-growth SaaS or PLG companies with large-scale event data.
• Experience with modern data stack including BigQuery, dbt, Fivetran, Segment, and Hex.
• Familiarity with experimentation platforms like LaunchDarkly, Statsig, or Eppo.
• Understanding of developer tools or software engineering workflows.
• Experience with causal inference methods like difference-in-differences, synthetic control, or CUPED.
• Familiarity with real-time data systems or operational analytics for monitoring agent performance.

Culture & Benefits

• Competitive Salary & Equity and 401(k) Program.
• Health, Dental, Vision, and Life Insurance, Short Term and Long Term Disability.
• Paid Parental, Medical, and Caregiver Leave.
• Commuter Benefits, Monthly Wellness Stipend, and In Office Set-Up Reimbursement.
• Flexible Time Off (FTO) + Holidays, Quarterly Team Gatherings, and In Office Amenities.
• Autonomous Work Environment.
• A commitment to building a diverse team representative of the world, welcoming unique perspectives and experiences.