Data Engineer

TL;DR

MTS – Data Engineering (AI): Building and scaling the data backbone that powers Copilot for millions of users worldwide with an accent on lightning-fast ETL pipelines and experimentation frameworks. Focus on solving complex data challenges and implementing robust schema management solutions.

Location: Must be local to the San Francisco area or Redmond area and in office 3 days a week.

Salary: USD $139,900 – $331,200 per year.

Company

Microsoft’s mission is to empower every person and every organization on the planet to achieve more.

What you will do

• Build, maintain, and enhance data ETL pipelines for processing large-scale data with low latency and high throughput to support Copilot operations.
• Design and maintain high throughput, low latency experimentation reporting pipelines.
• Own data quality initiatives including monitoring, alerting, validation, and remediation processes to ensure data integrity.
• Implement robust schema management solutions that enable quick and seamless schema evolution.
• Develop and maintain data infrastructure that supports real-time and batch processing requirements for machine learning model training and inference.
• Partner with cross-functional teams to understand data requirements and translate them into efficient technical solutions.

Requirements

• Master’s Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience OR Bachelor’s Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 6+ years experience OR equivalent experience.
• Experience building and maintaining production data pipelines at scale using technologies such as Apache Spark, Kafka, or similar distributed processing frameworks.
• Experience writing production-quality Python, Scala, or Java code for data processing applications.
• Experience with cloud data platforms (Azure, AWS, or GCP) and their data services.
• Experience with schema management and data governance practices.

Nice to have

• Experience building and scaling experimentation frameworks.
• Experience with real-time data processing and streaming architectures.
• Experience with data orchestration frameworks such as Airflow, Prefect, Dagster or similar workflow management systems.
• Experience with containerization technologies (Docker, Kubernetes) for data pipeline deployment.
• Demonstrated experience with data quality frameworks and monitoring solutions.

Culture & Benefits

• Growth mindset and innovation-focused environment.
• Collaborative culture with shared goals.
• Commitment to respect, integrity, and accountability.
• Emphasis on creating a culture of inclusion where everyone can thrive.