Principle Machine Learning Engineer (AI)

Location: Remote options; the team is based in Sydney and Melbourne, Australia.

Company

Studio is hirify.global's platform for customers to build, run, and manage AI agents under the Rovo brand.

What you will do

• Own the architecture for measuring agent quality, including production sampling, judgment methods, failure-mode analysis, and quality aggregation.
• Design cost- and accuracy-aware evaluation systems using deterministic checks, behavioral heuristics, and model-based judgment.
• Define benchmarks, datasets, evaluation environments, observability, reproducible experiments, and calibrated evaluators.
• Establish regression gates, pre-release checks, and production drift detection for agent, prompt, and model changes.
• Lead complex quality problems involving agent selection and routing, tool-use reliability, and autonomous agents running inside automations.
• Set the multi-quarter technical roadmap, partner with product, data science, central AI teams, and model providers, and mentor engineers through design reviews and hands-on work.

Requirements

• 6+ years of experience in ML/AI engineering, with deep expertise in LLMs, NLP, information retrieval, or recommendation.
• Proven experience building and shipping LLM-powered or agentic systems used by real customers and improving them through measurement.
• Strong evaluation and experimental skills, including benchmark design, assessment of label and judge quality, recognition of leakage and confounding factors, and connection of offline results to product impact.
• Strong software and data engineering skills, including validating designs in code and working with data pipelines, model serving, and production observability.
• Strong product judgment and the ability to define measurable outcomes and choose between deterministic, classical ML, and LLM-based approaches.
• Experience influencing technical direction across multiple teams and mentoring senior engineers.
• Regular use of AI tools in professional work and the ability to turn new research into production results.
• Sound judgment about trade-offs involving quality, latency, cost, reliability, privacy, and access control.

Nice to have

• Experience building agent platforms where customers configure the agents.
• Experience with trajectory-level evaluation of multi-step agents, or prompt and agent optimization.
• Experience in enterprise SaaS.
• A master's or PhD in Computer Science, Machine Learning, or a related field.

Culture & Benefits

• Remote work options are available.
• Work closely with product, data science, central AI teams, and model providers.
• Join a team of software and machine learning engineers working on applied ML in production agentic systems.