Staff Applied AI Scientist (LLMOps)

Location: Melbourne or Sydney, Australia; hybrid with an average of 2 days per week in a local hirify.global office. Candidates must be legally authorised to work in Australia for the duration of employment.

Company

hirify.global provides an employee experience platform that helps organisations improve engagement, performance management, and team development.

What you will do

• Own the feedback loop for Coach AI through prompt engineering, large-scale evaluation, and continuous product improvement.
• Build LLM-powered analysis tools that diagnose production performance shifts and recommend prompt or system-level changes.
• Design context engineering systems covering retrieval, memory, context assembly, compression, and context-budget management.
• Run evaluation programs using sampling, LLM-as-a-judge, de-identified production traces, human labelling, monitoring, and alerting.
• Improve agentic orchestration, model and provider routing, guardrails, PII handling, content safety, and jailbreak resistance.
• Create reusable frameworks, tooling, and documentation while partnering with AI, product, data science, and people science teams.

Requirements

• Experience building and operating production agentic systems, including context engineering, RAG, memory, model selection, cost, and performance optimisation.
• Hands-on experience with LLM evaluation, eval datasets, LLM-as-a-judge, human-in-the-loop labelling, thresholds, and longitudinal quality measurement.
• Experience with observability tooling for LLM and agentic systems, including traces, sampling, prompt management, and production monitoring such as Langfuse or comparable tools.
• Daily use of agentic coding tools such as Claude Code, Cursor, or Codex, with sound judgement about when to direct agents versus write code directly.
• Strong technical writing, communication, enablement, and mentoring skills, with experience teaching others to own capabilities.
• Legal authorisation to work in Australia is required.

Nice to have

• Experience scaling evaluation and observability practices across multiple teams.
• Experience evolving enterprise codebases with AI and building production agentic systems.
• Postgraduate degree in machine learning, computer science, applied mathematics, or a related field.
• Public writing, talks, or open-source work in evaluation, observability, or LLMOps.

Culture & Benefits

• Employee equity through the Employee Share Option Program.
• Learning programs, coaching, quarterly refresh days, and an extended end-of-year break.
• Monthly wellbeing and lifestyle allowance, inclusive parental leave from day one, and five annual social impact days.
• MacBook and budget for a home workspace.
• Medical insurance for employees and families in the US and UK.
• Hybrid office collaboration designed to support connection, pace, and shared culture.