Azure Data Engineer (Databricks)

What you will do

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Build ETL/ELT solutions for ingesting, transforming, and loading data from multiple sources.
• Develop and optimize PySpark applications and Spark SQL transformations for large-scale data processing.
• Create and manage Databricks notebooks, workflows, jobs, clusters, and Delta Lake solutions.
• Develop data ingestion frameworks with Azure Data Lake Storage Gen2 and monitor, troubleshoot, and optimize workloads.
• Implement CI/CD pipelines and collaborate with architects, analysts, and stakeholders on enterprise data solutions.

Requirements

• Strong programming experience with Python, PySpark, Spark SQL, and SQL.
• Hands-on experience with Azure Databricks, Azure Data Factory, and Azure Data Lake Storage Gen2.
• Experience developing and optimizing ETL/ELT solutions and modeling data.
• Knowledge of Delta Lake, Databricks Workflows, notebook development, and Spark performance tuning.
• Experience with Git or GitHub and an understanding of data quality, security, governance, and compliance.
• Ability to work as an overseas contractor and collaborate with data architects, business analysts, and stakeholders.

Nice to have

• Azure Synapse Analytics, Unity Catalog, Azure DevOps CI/CD, and Azure Key Vault.
• Snowflake and Azure monitoring and logging solutions.
• Experience with data governance and data quality frameworks, CDC, and real-time data processing.
• Microsoft Azure Data Engineer Associate DP-203 or Databricks Data Engineer Associate certification.