2 дня назад
Senior Data Engineer (Databricks)
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Senior Data Engineer (Databricks): Designing, building, and operating scalable data pipelines and end-to-end data workflows on Databricks with an accent on PySpark, Delta Lake, data quality, and legacy code modernization. Focus on optimizing Spark jobs and cluster configurations, migrating ETL processes to ELT patterns, and maintaining reliable production pipelines.
Location: Singapore; onsite. Travel to Singapore is required within 30–45 days of onboarding, subject to paperwork completion.
Company
provides IT consulting services and places technical specialists on client projects.
What you will do
- Design, build, operate, and optimize scalable data pipelines and end-to-end workflows on Databricks.
- Develop ingestion, transformation, and consumption processes using PySpark, Spark SQL, and the DataFrames API.
- Implement error handling, monitoring, alerting, data quality checks, and validation frameworks.
- Modernize legacy code and migrate traditional ETL processes to ELT patterns while preserving data integrity and backward compatibility.
- Design and maintain Delta Lake tables, reusable data engineering frameworks, and production data models.
- Collaborate with architects, analysts, business stakeholders, Infrastructure, Applications, and Cyber teams; mentor junior data engineers.
Requirements
- At least one Databricks certification is required: Databricks Certified Data Engineer Associate or Professional.
- At least 5 years of experience in data engineering or related roles; the professional experience section specifies a minimum of 8 years.
- 2–3 years of hands-on Databricks experience, including workspace, cluster management, notebooks, and job orchestration.
- Strong Python and PySpark skills, including DataFrames API, Spark SQL, performance optimization, and production pipeline development.
- Experience with data modeling, Delta Lake, multiple data sources and formats, SQL, cloud platforms, data governance, security, testing, Git, and CI/CD.
- Excellent communication and stakeholder management skills, with experience in agile development environments.
Nice to have
- Databricks Certified Associate Developer for Apache Spark.
- Azure, AWS, or Google Cloud data engineering certifications.
- Experience with Structured Streaming and Databricks Workspace AI Agent integration.
Culture & Benefits
- Work across data, infrastructure, applications, cybersecurity, and business stakeholder groups.
- Participate in code reviews and share engineering practices with Team NCS.
- Document technical solutions, refactoring approaches, and migration playbooks.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →