Master Data Developer (Databricks)

Location: Brazil; hybrid workplace.

Company

CI&T delivers AI deployment, application modernization, data and AI, Martech, and business strategy solutions for large enterprises.

What you will do

• Build, optimize, and maintain scalable data pipelines and ETL/ELT processes for analytics and business intelligence.
• Implement data lakes, warehouses, integration layers, and other data architectures.
• Implement data quality checks, validation processes, monitoring, security practices, and data privacy compliance.
• Develop integrations between data sources, third-party tools, APIs, and analytics platforms.
• Monitor pipeline performance, query efficiency, and storage costs while maintaining technical documentation.
• Collaborate with Data & Analytics Managers and business stakeholders, using AI tools and automation to improve data engineering workflows.

Requirements

• Hands-on experience with Databricks migration using the databricks/migrate tool, including notebooks, jobs, clusters, secrets, ACLs, Unity Catalog, and Glue Catalog.
• Experience migrating notebooks, DAG pipelines, and DLT pipelines.
• Experience refactoring /mnt/ and DBFS references to S3-backed storage locations.
• Experience executing data migrations from Delta Lake to Iceberg on S3, including Delta tables, raw files, and incremental synchronization.
• Experience with streaming migration, including Auto Loader, DLT checkpoint re-bootstrap, and event ID deduplication.
• Experience validating migrations through row counts, checksums, sample queries, and ETL runtime comparisons.
• Experience recreating secret scopes in AWS Secrets Manager.
• Knowledge of AWS S3, Glue, Secrets Manager, IAM basics, Athena, Lake Formation, and EMR.
• Knowledge of Spark SQL, PySpark, Java, Python, and Scala.
• Experience working in an Agile pod with daily ticket updates, standups, and code review.

Nice to have

Culture & Benefits

• Hybrid work arrangement in Brazil.
• Collaboration with teams across more than 25 countries.
• Benefits are provided as part of the CI&T employment offering.

Hiring process

• Present an assessment during the selection process.
• Support is available for obtaining the assessment if needed.