2 дня назад
Senior Data Engineer (Azure Databricks)
Вакансия из Hirify Global, списка международных tech-компанийДля мэтча и отклика нужен Plus
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Senior Data Engineer (Azure Databricks): Building and maintaining scalable ETL/ELT pipelines and governed Delta Lake data products across medallion architecture layers with an accent on Spark/PySpark, Unity Catalog, data quality, and cross-cloud integration. Focus on designing observable pipelines, preparing AI/ML-ready datasets, optimizing lakehouse performance, and automating delivery through Databricks Workflows and CI/CD.
Location: Bangalore, India — onsite
Company
BTVK Advisory is an advisory firm providing business and professional services through operations across North America, South America, Europe, Asia, and Australia.
What you will do
- Design, build, and maintain scalable ETL/ELT pipelines on Azure Databricks across bronze, silver, gold, and platinum medallion layers.
- Ingest structured, semi-structured, and unstructured data using Spark Structured Streaming, Auto Loader, JDBC/CDC, Lakeflow Connect, Databricks Marketplace, and Delta Sharing.
- Build Unity Catalog assets and governed integrations with ADLS Gen2, including access controls, lineage, data classification, PII handling, and retention.
- Design audit, control, metadata, observability, reconciliation, and data-quality frameworks for reliable pipeline operations.
- Develop performant Delta Lake models and AI/ML-ready datasets supporting feature engineering, feature stores, embeddings, vector data, analytics, Power BI, and Tableau.
- Orchestrate workflows with Databricks Jobs, Workflows, and Lakeflow Declarative Pipelines, using Git and Databricks Asset Bundles for CI/CD; troubleshoot issues and collaborate with BI, analytics, and business teams.
Requirements
- Bachelor’s degree in Information Technology, Computer Science, Data Science, Analytics, Statistics, or a related field.
- 5–8 years of experience in data engineering, data integration, or analytics engineering, including substantial Databricks experience.
- Hands-on expertise with Azure Databricks workspaces, clusters, SQL warehouses, Unity Catalog, and ADLS Gen2.
- Strong proficiency in Spark/PySpark and SQL, including complex queries and advanced data transformations.
- Experience with Delta Lake medallion architecture, streaming ingestion, cross-cloud data migration or sharing, orchestration, CI/CD, governance, and data-quality frameworks.
- Strong analytical, problem-solving, documentation, testing, code-review, and Agile delivery skills.
Nice to have
- Experience with tax, assurance, and financial data and its confidentiality and regulatory requirements.
- Exposure to MLflow, feature stores, Mosaic AI, Vector Search, or other Databricks AI/ML capabilities.
- Experience with Power BI or Tableau data modeling and semantic-layer design.
- Databricks Certified Data Engineer Professional certification.
Culture & Benefits
- Full-time employment with an advisory organization operating across multiple global regions.
- Work is delivered using Agile or hybrid project methodologies.
- Equal opportunity employment practices apply.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →