Machine Learning Engineer (Public Sector)

TL;DR

Machine Learning Engineer (Public Sector): Develop and scale automated evaluation pipelines for advanced AI models including LLMs in mission-critical government environments with an accent on model reliability, safety, and robustness. Focus on designing evaluation frameworks, stress testing, and continuous monitoring to ensure trustworthy AI deployment in defense and federal missions.

Location: San Francisco, CA; St. Louis, MO; New York, NY; Washington, DC, USA

Salary: $187,000–$300,000 USD

Company

Scale AI develops reliable AI systems and full-stack technologies powering leading models for enterprises and governments, including defense and federal agencies.

What you will do

• Develop and maintain automated evaluation pipelines for ML models across performance, robustness, and safety metrics.
• Design test datasets and benchmarks to measure generalization, bias, explainability, and failure modes.
• Build evaluation frameworks for LLM agents including scenario-based testing infrastructure.
• Conduct comparative analyses of model architectures and training procedures.
• Implement continuous monitoring, regression testing, and quality assurance tools.
• Design and execute stress tests and red-teaming workflows to identify vulnerabilities.

Requirements

• Must be able to obtain or hold an active security clearance.
• Experience in computer vision, deep learning, reinforcement learning, or NLP in production.
• Strong programming skills in Python and experience with TensorFlow or PyTorch.
• Background in algorithms, data structures, and object-oriented programming.
• Experience with LLM pipelines, simulation environments, or automated evaluation systems.
• Ability to translate research insights into measurable evaluation criteria.

Nice to have

• Graduate degree in CS, ML, or AI.
• Cloud experience (AWS, GCP) and model deployment experience.
• Experience with LLM evaluation, CV robustness, or RL validation.
• Knowledge of interpretability, adversarial robustness, or AI safety frameworks.
• Experience in regulated, classified, or mission-critical ML domains.

Culture & Benefits

• Comprehensive health, dental, and vision coverage.
• Retirement benefits and equity compensation.
• Learning and development stipend.
• Generous paid time off.
• Commuter stipend and inclusive workplace.