Data Engineer (Databricks)

TL;DR

Data Engineer (Databricks): Building and managing robust data pipelines and infrastructure for a healthcare platform with an accent on Spark optimization and MongoDB integration. Focus on designing scalable data workflows, automating processing tasks, and ensuring high performance for large-scale distributed systems.

Location: On-site in Vancouver

Company

A healthcare technology company partnering with specialty clinics to optimize clinical workflows and patient engagement through a data-driven platform.

What you will do

• Develop, manage, and optimize data pipelines on the Databricks platform.
• Debug and troubleshoot Spark applications to ensure reliability and performance.
• Write clean, efficient Python code and design APIs for data integration.
• Manage and optimize MongoDB data storage and retrieval processes.
• Collaborate with data scientists and analysts to deliver scalable data solutions.
• Identify and resolve technical challenges in system design and data processing.

Requirements

• Proven experience working with Databricks and Spark compute.
• Proficiency in Python, including object-oriented programming and API development.
• Familiarity with NoSQL databases, specifically MongoDB.
• Experience with large-scale data processing and distributed systems.
• Strong problem-solving and debugging skills.
• Must be able to work on-site in Vancouver.

Nice to have

• Understanding of data architecture, ETL processes, and data warehousing.
• Knowledge of Delta Lake, Hadoop, or Kafka.
• Experience with cloud platforms like AWS, Azure, or GCP.
• Familiarity with CI/CD pipelines and Git.

Culture & Benefits

• Ownership-driven environment where team members take responsibility for outcomes.
• Fast-paced startup culture that values speed and rapid execution.
• Commitment to personal and professional growth through continuous learning.
• Direct, constructive feedback culture.
• High attention to detail and focus on delivering accurate, high-quality results.