Staff AI Engineer - QA
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Location: Remote within the United States or Canada; listed U.S. locations include North Carolina, South Carolina, California, New York, New Mexico, Michigan, Louisiana, Virginia, Utah, Georgia, Texas, Florida, and Tennessee (CST)
Salary: $120,000–$160,000 annually, depending on experience and qualifications.
Company
provides mission-critical software solutions for public sector, healthcare, utilities, and private-sector organizations across North America, Europe, Asia, and Australia.
What you will do
- Build and own the architecture of the AI system end to end, including service layers, data pipelines, and evaluation infrastructure.
- Design dataset management, automated test harnesses, and historical workflow replay for AI evaluation.
- Maintain clinician-reviewed evaluation datasets and develop severity-weighted error metrics for clinical and financial impact.
- Build production monitoring for accuracy, overrides, and safety issues with actionable alerts.
- Develop adversarial tests for hidden or malicious document instructions and invented content.
- Partner with product and clinical leads to define accuracy thresholds and criteria for expanding or withdrawing AI capabilities.
Requirements
- 7+ years of experience in software or machine learning engineering, including at least 3 years focused on AI evaluation, testing, or quality.
- Experience designing and building complete systems from prototype through production.
- Experience building and operating production evaluation systems used by engineering teams.
- At least 2 years evaluating large language model systems in production.
- Strong statistical skills, including sampling, confidence intervals, and error analysis.
- 1+ year designing evaluation sets for document extraction, classification, or structured output.
- Deep knowledge of explainable AI techniques and AI-powered tools such as SHAP, LIME, and Claude.
- Clear written communication, including design documents, and the ability to explain complex trade-offs to clinicians, engineers, and executives.
- Curiosity and willingness to adopt emerging AI tools, including copilots, chat-based AI, and automation solutions, in everyday work.
Nice to have
- Experience in healthcare or another highly regulated industry.
- Familiarity with metamorphic testing, adversarial testing, or data validation tools such as Great Expectations.
- Experience mentoring engineers and influencing technical direction across multiple teams.
Culture & Benefits
- Full-time remote work with flexible work arrangements.
- Medical, dental, and vision benefits.
- Three weeks of vacation plus five personal days.
- Employee stock ownership, RRSP program, and 401(k) matching.
- Career growth, learning opportunities, community involvement, and a casual work environment.
Будьте осторожны
Если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого. Это мошенники. Обязательно жмите «Пожаловаться» или пишите в поддержку. Подробнее в гайде →