Senior AI Engineer - CD Visibility

TL;DR

Senior AI Engineer (AI): Building and optimizing AI-driven decision systems for CI/CD pipelines to detect and block faulty changes, with an accent on combining Datadog telemetry for explainable outcomes and designing autonomous remediation. Focus on robust and safe AI system design, owning evaluations and rollout processes, and integrating across the Datadog platform.

Location: Madrid, Spain. Hybrid workplace

Company

Datadog is a global SaaS business delivering growth and profitability through its observability platform for cloud applications.

What you will do

• Build AI-driven deployment gates for CI/CD, producing safe, explainable allow/block outcomes.
• Own evaluations and rollout processes, defining metrics, and validating changes in shadow mode.
• Design for robustness and safety, implementing conservative defaults, guardrails, and fallbacks.
• Partner closely with Product to translate customer problems into technical decisions.
• Integrate with internal AI teams on faulty deployment detection, LLMs, and AI agents.
• Own and operate reliable backend services that run in the critical path of customer deployments, including on-call duties.

Requirements

• A Product-minded engineer who ships AI to production.
• 5+ years experience with backend systems and microservices performance.
• Proficiency in a modern programming language; strong API/service design; production ops.
• Proven experience delivering LLM/agent features to production.
• Comfortable owning user journeys, iterating from prototype to GA, and measuring impact with clear product metrics.
• Understands the end-to-end LLM product lifecycle and is fluent with offline/online evals for AI systems.

Nice to have

• Experience with Continuous Delivery tools (i.e. ArgoCD / Argo Rollouts, Spinnaker, Octopus Deploy).
• Exposure to planning/agent frameworks, tool-use orchestration, RAG, and retrieval/indexing for observability data.

Culture & Benefits

• New hire stock equity (RSUs) and employee stock purchase plan (ESPP).
• Continuous professional development, product training, and career pathing.
• Intradepartmental mentor and buddy program for in-house networking.
• An inclusive company culture and ability to join Community Guilds.
• Free, global mental health benefits for employees and dependents age 6+.
• Competitive global benefits (may vary based on country of employment).