Senior AI Engineer (APM)

TL;DR

Senior AI Engineer (APM): Building and optimizing LLM- and Agent-based features for Application Performance Monitoring to detect, resolve, and prevent performance issues with an accent on autonomous agents and diagnostic workflows. Focus on designing robust agent systems, implementing retrieval and planning strategies, and productionizing scalable AI capabilities within the Datadog platform.

Location: Hybrid in New York, NY, USA

Salary: $187,000 — $240,000 USD

Company

Datadog is a leading observability and security platform providing cloud-scale monitoring and analytics for software engineers.

What you will do

• Design and ship LLM/agentic workflows that analyze telemetry to generate diagnoses and guided fixes.
• Own the full product loop from rapid prototyping and success metrics to production-scale reliability.
• Develop robust agent systems, including retrieval and planning strategies, tool-use, and safety guardrails.
• Integrate AI features into Datadog’s core platform surfaces like Trace Explorer and Service Catalog.
• Collaborate with PM and Design teams to build cohesive, product-minded AI experiences.
• Write performant backend code and maintain high-throughput, low-latency data systems in production.

Requirements

• 4+ years of experience building backend or real-time ML systems.
• Proven experience delivering LLM/agent features to production, including prompting and evals.
• Strong fundamentals in the ML lifecycle and statistical experiment design.
• Proficiency in Go, Java, or Python with strong API and service design skills.
• Experience with microservices performance, including tracing, latency, and resiliency patterns.
• Must be based in New York, NY.

Nice to have

• Hands-on experience with distributed tracing stacks like OpenTelemetry or Datadog APM.
• Exposure to RAG, tool-use orchestration, and retrieval/indexing for observability data.
• Familiarity with SLO/SLA practices and incident response.

Culture & Benefits

• Opportunity to build high-impact tools for other software engineers.
• Significant influence on product direction and business growth.
• Collaborative environment with a culture of continuous professional development.
• Comprehensive benefits including 401(k) match, healthcare, dental, and parental planning.
• Paid time off and fitness reimbursements.