Senior Software Engineer (AI)

TL;DR

Senior Software Engineer (AI): Building AI-driven deployment gates and analysis services to prevent production incidents with an accent on LLM integration and autonomous remediation. Focus on designing explainable decision systems, implementing offline/online evaluations, and scaling reliable backend services.

Location: Hybrid in Madrid, Spain

Company

Datadog is a cloud-scale observability platform providing monitoring and security for cloud applications.

What you will do

• Build AI-driven deployment gates using CI/CD context and telemetry to produce safe, explainable outcomes.
• Own the end-to-end evaluation process, defining precision/recall metrics and validating changes in shadow mode.
• Design robust safety guardrails and human-in-the-loop paths to ensure predictable AI behavior.
• Collaborate with Product Managers to translate customer problems and roadmap goals into technical iterations.
• Partner with internal AI teams to integrate Faulty Deployment Detection and LLM agent frameworks.
• Build and operate reliable backend services in the critical path of customer deployments.

Requirements

• 5+ years of experience with backend systems and microservices (tracing, latency, concurrency, and resiliency).
• Proven track record of shipping LLM/agent-based features to production.
• Proficiency in a modern programming language with strong API/service design and production ops skills.
• Experience with end-to-end LLM product lifecycles and offline/online evaluation frameworks.
• Must be based in or able to work hybrid in Madrid, Spain.

Nice to have

• Experience with Continuous Delivery tools such as ArgoCD, Spinnaker, or Octopus Deploy.
• Exposure to RAG, retrieval/indexing for observability data, and agent orchestration frameworks.

Culture & Benefits

• Equity packages including RSUs and an Employee Stock Purchase Plan (ESPP).
• Hybrid work model designed to balance office collaboration with work-life harmony.
• Global mental health benefits for employees and their dependents.
• Continuous professional development, internal mentor programs, and community guilds.