Applied AI Software Engineer (Medtech)

TL;DR

Applied AI Software Engineer (AI Engineering): Developing and evaluating LLM-based agents for healthcare automation with an accent on rigorous benchmarking, safety, and reliability in clinical contexts. Focus on building end-to-end evaluation pipelines, defining ground truth outcomes, and optimizing model configurations for production deployment.

Location: San Francisco, CA / Remote (US)

Company

hirify.global is an electronic medical records (EMR) and payments development platform for healthcare.

What you will do

• Design and execute large-scale evaluation plans for LLM-based agents performing clinical documentation, scheduling, billing, and workflow automation.
• Build end-to-end test harnesses to validate model behavior across different prompt templates, context sources, and tool configurations.
• Collaborate with clinicians and subject matter experts to define gold-standard expected outcomes for performance comparisons.
• Run and replicate experiments across multiple foundation models and parameters to determine optimal configurations.
• Take ownership of internal evaluation tooling and infrastructure to ensure speed and reproducibility.
• Identify and recommend candidates for reinforcement fine-tuning or retrieval augmentation based on evaluation gaps.

Requirements

• 5+ years of experience in applied machine learning or AI engineering, with a focus on evaluation and benchmarking.
• Proficiency with foundation model APIs (OpenAI, Claude, Gemini) and experience orchestrating complex agent behaviors.
• Superlative Python engineering skills and familiarity with SQL and database management.
• Experience designing high-throughput evaluation pipelines, including human-in-the-loop or expert-labeled benchmarks.
• Ability to collaborate across engineering, product, and clinical subject matter experts.

Nice to have

• Familiarity with clinical or healthcare data.
• Experience with reinforcement fine-tuning, model monitoring, or RLHF.

Culture & Benefits

• Institutionally backed by top-tier technology investors.
• Opportunity to solve complex healthcare challenges using state-of-the-art AI.
• Collaborative environment working with ML engineering and clinical informatics teams.
• Equal employment opportunity environment.