Machine Learning Eval Engineer (AI)

TL;DR

Machine Learning Eval Engineer (AI): Building evaluation systems and benchmarks to identify model failure modes and improve model quality with an accent on unstructured enterprise data processing. Focus on developing metrics, heuristics, and tooling for automated model assessment at scale.

Location: Must be based in San Francisco (In-person role)

Salary: $150,000 – $300,000

Company

hirify.global helps AI teams ingest and process unstructured enterprise data like PDFs and spreadsheets using vision models and LLMs.

What you will do

• Design, build, and maintain evaluation benchmarks to reveal model performance and failure modes.
• Develop metrics and workflows to automatically identify issues across large real-world datasets.
• Partner with ML engineers to turn evaluation insights into model improvements and training priorities.
• Work hands-on with complex document formats including PDFs and spreadsheets.
• Build internal tooling using Python to help teams analyze model behavior and inspect results.
• Collaborate with customers to create bespoke benchmarks reflecting real-world data needs.

Requirements

• Strong Python skills with the ability to build clean and reliable technical solutions independently.
• Experience working with data infrastructure such as AWS S3 and analytics systems like Tinybird.
• High bar for quality and precision with a focus on solving complex problems from first principles.
• Effective collaboration across both technical and non-technical teams.
• Ability to take full ownership from strategy through execution in a fast-paced environment.
• Must be able to work in person at our San Francisco office.

Nice to have

• Experience in an early-stage or high-growth startup.
• Background in product thinking and building user-facing interfaces.
• Direct experience working with customers to understand their workflows.
• Experience in document understanding systems or enterprise software.

Culture & Benefits

• Comprehensive health insurance coverage including medical, dental, and vision.
• Unlimited PTO policy.
• Monthly health and wellness reimbursement of $150.
• Daily free lunch provided at the office.
• Reimbursed transportation costs.
• Supportive parental leave policies.