Core Speech AI Evaluation Engineer (AI)

Location: Hybrid in Pittsburgh, Pennsylvania, United States; remote/hybrid work is stated, with business-hours on-call coverage, up to 5% domestic/international travel, and required on-site onboarding at a designated company location. Relocation assistance may be authorized. Candidates must be legally authorized to work in the United States without employment visa sponsorship.

Expected compensation: $106,000–$145,750, including base pay and variable incentive pay if eligible.

Company

hirify.global develops healthcare solutions at the intersection of health, materials, and data science, including products that support healthcare professionals and patients.

What you will do

• Lead quality benchmarks and evaluation processes for AI-produced medical notes, including automated LLM-as-judge evaluation.
• Implement data analysis pipelines that produce statistical quality reports for research and development and support teams.
• Support actionable analytics for NLP and NLU products.
• Collaborate with global teams on products supporting English, German, Spanish, and Italian.
• Participate in a business-hours restricted on-call rotation to resolve team tool errors.

Requirements

• Bachelor’s degree or higher and 4 years of data analytics experience, or a high school diploma/GED and 8 years of data analytics experience.
• Experience in AI-model quality analysis.

Nice to have

• Experience in linguistics and/or human-computer interaction.
• Experience with natural language processing or natural language understanding.
• Experience with medical or clinical software development.
• Experience in evaluation design, including rater training, blinding, inter-rater agreement, sample sizing, and identifying noise.
• Specific experience in AI-model quality and safety analysis.
• Ability to work across team boundaries toward a shared goal.

Culture & Benefits

• Hybrid work arrangement with company-supported on-site onboarding for new hires.
• Travel arrangements and related onboarding expenses are coordinated and paid by the company according to its travel policy.
• Benefits may include medical, dental and vision coverage, health savings accounts, flexible spending accounts, disability benefits, life insurance, paid absences, and retirement benefits.
• Work supports healthcare products used by professionals and patients.