Senior Machine Learning Engineer (AI)

Location: Remote options; the team is based in Sydney and Melbourne.

Company

hirify.global develops Rovo, a model-agnostic cloud agent platform integrated with Jira, Confluence and automation workflows.

What you will do

• Build offline evaluation suites and benchmarks for realistic Rovo agent tasks using usage patterns, synthetic scenarios and simulated tools.
• Develop online quality metrics for production traffic, including sampling strategies, behavioral signals and model-based evaluation of agent trajectories.
• Calibrate model judges against human ratings and independent outcome signals, and evolve evaluations as agents improve.
• Improve agent quality across prompts, context selection, tool use, retrieval and agent routing.
• Reduce cost and latency across production LLM systems.
• Own projects from problem definition through experimentation, deployment and ongoing measurement; contribute to reviews and coach engineers on writing effective evaluations.

Requirements

• Bachelor’s or master’s degree, preferably in Computer Science or a related field, or equivalent practical experience.
• 4+ years of related industry experience in machine learning, including hands-on experience designing, evaluating and operating LLM-based or agentic systems.
• Strong coding skills in Python and/or Kotlin, with the ability to write performant, production-quality code using AI.
• Experience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior to guide improvements.
• Familiarity with SQL, distributed data processing and cloud data environments.
• Experience building and operating ML systems in production, including practices for improving quality, reliability and cost.
• Regular use of AI tools in daily engineering work, with the judgment to validate outputs and take ownership of results.
• Ability to lead ambiguous technical work from problem definition through delivery, with a strong ownership mindset.
• Clear communication skills and the ability to explain ML trade-offs to engineers, product managers and leaders.

Nice to have

• Experience developing agent evaluations, including trajectory analysis, human evaluation or LLM-as-a-judge.
• Experience building agents that use tools or retrieval-augmented generation applications.
• Experience with retrieval, ranking or recommendation.
• Experience in enterprise SaaS or similarly large-scale product environments.

Culture & Benefits

• Remote options are available.
• Work with software and machine learning engineers based in Sydney and Melbourne.
• Collaborate closely with product, data science and central AI teams.
• Build applied machine learning systems for real agentic products in production.