Member of Technical Staff, Evaluations (AI)

Location: New York, NY; offices are also available in San Francisco and Los Angeles. Relocation support is provided. Employment may require authorization to access technology controlled under U.S. export-control laws without sponsorship for an export license.

Salary: $175,000–$250,000 annual base salary, plus equity and benefits.

Company

hirify.global builds Atlas, an AI platform that uses physics-grounded intelligence to verify, debug, and optimize hardware across its lifecycle.

What you will do

• Own the design and reliability of evaluation harnesses and build automated reporting pipelines for models and tools.
• Design ablations to measure the impact of changes to models, harnesses, tools, and context on performance, cost, and runtime.
• Analyze agent trajectories to identify failure modes, performance clusters, and root causes such as context loss, tool inefficiency, and engineering gaps.
• Build improvement loops for skills and harnesses across electrical and broader hardware-engineering tasks.
• Develop verifiable scoring systems and calibrate LLM judges against expert-engineer assessments.
• Partner with Operations, Electrical Engineering, customers, and technical partners to create synthetic datasets and translate domain expectations into measurable evaluations.

Requirements

• Strong Python skills for reliable research or production infrastructure, data pipelines, and tooling.
• Experience designing evaluations, benchmarks, or metrics for ML systems, preferably large language models or agentic systems.
• Hands-on experience with LLM agents, harnesses, tool use, and agent-trajectory analysis.
• Strong grounding in experimental design, statistics, and trustworthy metric development.
• Excellent written and verbal communication for explaining technical results to specialists and non-specialists.
• Ability to work in the United States and meet applicable U.S. export-control authorization requirements.

Nice to have

• Electrical or hardware-engineering experience, including circuit design, RF/EM, signal integrity, PCB layout, or test and measurement.
• Experience calibrating and validating LLM-as-judge systems.
• Experience sourcing or hand-crafting synthetic datasets and evaluating multimodal, spatial, or spatiotemporal reasoning.
• Experience building trusted dashboards and reports or working directly with customers and external technical partners.

Culture & Benefits

• Work with a 60-person team combining AI engineering and applied-physics expertise.
• Medical, vision, and dental insurance premiums covered for employees and dependents.
• 401(k) retirement plan and unlimited PTO.
• Daily lunch from local restaurants.
• Relocation support.