Senior Data Engineer (Databricks)

TL;DR

Senior Data Engineer (Databricks): Designing and implementing scalable data pipelines and platforms for international clients using the Databricks Lakehouse Platform with an accent on Apache Spark and Delta Lake. Focus on optimizing batch and streaming workflows, improving processing cost-efficiency, and ensuring high data quality.

Location: Hybrid; must be based in Japan

Company

A global data and AI consultancy specializing in Big Data, Cloud, Streaming, and Microservices for major international enterprises.

What you will do

• Design, implement, and maintain scalable data pipelines using the Databricks Lakehouse Platform, Apache Spark, and Delta Lake.
• Develop and optimize batch and streaming data workflows for analytics, machine learning, and BI applications.
• Collaborate with cross-functional teams to translate business requirements into production-ready data solutions.
• Improve performance and cost efficiency of data processing workloads within Databricks.
• Implement best practices for data quality, governance, and security.
• Mentor junior engineers and contribute to internal knowledge sharing.

Requirements

• Strong experience with Databricks and Apache Spark (PySpark, SQL, or Scala).
• Experience building data platforms on major cloud providers (AWS, Azure, or GCP).
• Solid understanding of distributed data processing, data modeling, and performance optimization.
• Experience with CI/CD pipelines and version control tools such as Git.
• Familiarity with orchestration frameworks like Airflow or Databricks Workflows.
• Strong communication skills in English and Japanese for collaboration in an international environment.

Culture & Benefits

• Competitive salary aligned with Japanese market standards.
• Hybrid work options combining office and remote work.
• Exposure to large-scale data and AI projects for global organizations.
• Professional development opportunities, including certifications and Udemy Business.
• Structured performance reviews and regular feedback cycles.
• Collaborative, knowledge-driven culture with global team events.