Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks): Designing and implementing distributed big data solutions and Lakehouse architectures with an accent on ETL pipelines, Spark optimization, and model deployment. Focus on architecting scalable data platforms, leading client migrations, and optimizing cluster configurations for enterprise-scale implementation.

Location: Remote from Colombia, Argentina, Brazil, Georgia, Poland, or Ukraine

Company

AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence.

What you will do

• Provide technical leadership and guide customers through the implementation of big data projects.
• Architect and design Databricks-based solutions from concept to production.
• Develop end-to-end ETL pipelines, including streaming and batch processing.
• Lead client migrations and implement Lakehouse architectures.
• Mentor team members and provide technical support for pre-sales efforts.

Requirements

• Experience leading the design and development of distributed Databricks-based solutions.
• Proficiency in Python and expert knowledge of SQL and Spark/PySpark.
• Hands-on experience using dbt within the Databricks ecosystem.
• Solid understanding of data governance, security, and compliance.
• Experience in platform optimization, including cluster configuration and partitioning strategies.
• Bachelor's degree in Computer Science, Engineering, or equivalent work experience.

Nice to have

• Databricks Certification.
• Experience with DevOps using Terraform.
• Deployment of scalable ML pipelines using MLflow and Delta Lake.
• Experience with Azure, AWS, or GCP Databricks deployments.
• Proficiency in Scala or experience with BI tools like Power BI.

Culture & Benefits

• Opportunity to shape real-world AI-driven projects across diverse industries.
• Collaboration within a global team across multiple continents and cultures.
• Inclusive environment prioritizing continuous learning and innovation.
• Commitment to high ethical AI standards.