Principal Software Engineer, Applied AI

Location: San Francisco, California; in-person onboarding is required.

Annual base salary: $224,000–$308,000 USD for candidates located in the San Francisco Bay Area. Equity, bonus, and benefits may also apply.

Company

Okta builds identity and security infrastructure, including autonomous AI agents for identity operations and enterprise security.

What you will do

• Build agent harnesses for planning, tool use, memory, context compaction, interruption, and recovery during long-running workflows.
• Integrate OpenAI and Anthropic model APIs and select models based on quality, latency, cost, and failure handling.
• Connect agent harnesses to AWS workflow execution and approval state while keeping sensitive writes behind application policy checks.
• Create evaluation datasets, replay tests, and regression tests for findings, access decisions, application setup, and AI agent onboarding.
• Measure false positives, missed risks, unsafe actions, completion rates, and cost; turn production failures into improvements.
• Improve prompts, retrieval, tools, model routing, and post-training approaches while protecting tenant boundaries.

Requirements

• Strong Python engineering skills and experience shipping LLM applications that use tools.
• Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.
• Experience with model APIs, structured outputs, asynchronous services, and OpenAI or Anthropic integrations.
• Depth in agent execution or evaluation systems, including working code and explainable failure analysis.
• Ability to distinguish plausible model responses from supported findings or verified actions.
• Principal-level ownership of product engineering direction and shared components.

Nice to have

• Open-source contributions, agent observability, security evaluations, or synthetic evaluation data.
• Experience with fine-tuning, preference data, post-training evaluation, or model distillation.
• Experience with identity and security workflows.
• Foundation-model research experience.

Culture & Benefits

• Startup-level ownership and pace with the resources of a public company.
• Work on products intended for Okta's 20,000+ customers.
• Health, dental, and vision insurance, 401(k), flexible spending account, PTO, and parental leave.
• In-person onboarding designed to support connection and accelerate impact.
• Global community spanning more than 20 offices.