Databricks Engineer (Azure)

TL;DR

Databricks Engineer (Azure): Designing, implementing, and optimizing large-scale data pipelines for financial services and technology firms, ensuring scalability, reliability, and performance. Focus on leveraging PySpark/Spark and SQL for data transformation, integrating diverse data sources, and managing data governance and security within an Azure environment.

Location: Flexible hybrid policy in Mississauga, ON, Canada

Salary: CAD $110,000 – CAD $125,000/year

Company

hirify.global is a global consulting firm specializing in digital transformation for financial services and technology firms, with a strong focus on AI, Cloud, Data, and Software Engineering solutions.

What you will do

• Build and maintain scalable ETL/ELT data pipelines using Databricks, PySpark/Spark, and SQL.
• Integrate data from various sources including Azure Blob Storage and ADLS.
• Collaborate with cross-functional teams to understand business needs and deliver tailored data solutions.
• Optimize Databricks workloads for cost efficiency and performance, monitoring and troubleshooting pipelines.
• Implement and manage data security, access controls, and governance standards using Unity Catalog.
• Leverage Databricks Asset Bundles for seamless deployment and manage version control with GitLab.

Requirements

• Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture).
• Proficiency in Azure Cloud Services.
• Solid understanding of Spark and PySpark for big data processing.
• Experience with relational databases.
• Knowledge of Databricks Asset Bundles and GitLab.

Nice to have

• Familiarity with Databricks Runtimes and advanced configurations.
• Knowledge of streaming frameworks like Spark Streaming and experience in real-time data solutions.
• Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification.

Culture & Benefits

• Opportunity to work in a multinational organization with global offices.
• 15 days paid annual leave plus 10 personal leave days.
• Comprehensive insurance plan including medical, dental, vision, life, and long-term disability.
• Flexible hybrid work policy.
• RRSP with employer’s contribution up to 4%.
• Higher education certification policy and free access to Udemy for Business.
• Coaching opportunities with experienced colleagues from FinLabs and CoE groups.
• Cutting-edge projects at leading financial institutions.