Master Data Developer (Databricks)

Location: Colombia; remote.

Company

CI&T delivers AI deployment, application modernization, data and AI, Martech, and business transformation solutions for large enterprises.

What you will do

• Build, optimize, and maintain scalable data pipelines and ETL/ELT processes for analytics and business intelligence.
• Implement data lakes, warehouses, integration layers, data quality checks, validation processes, and monitoring systems.
• Develop integrations between data sources, third-party tools, APIs, and analytics platforms.
• Collaborate with Data & Analytics Managers and business stakeholders to translate requirements into technical solutions.
• Optimize pipeline performance, query efficiency, and storage costs while documenting architectures, pipelines, and processes.
• Apply data security and privacy best practices and use AI tools and automation to improve data engineering workflows.

Requirements

• Hands-on experience migrating Databricks assets with the databricks/migrate tool, including notebooks, jobs, clusters, secrets, and ACLs.
• Experience migrating Unity Catalog to Glue Catalog, including notebook, DAG pipeline, and DLT pipeline migration.
• Experience refactoring /mnt/ and DBFS storage references to S3-backed locations.
• Experience migrating Delta Lake data to Iceberg on S3, including Delta tables, raw files, and incremental synchronization.
• Streaming experience with Auto Loader, DLT checkpoint re-bootstrap, and event_id deduplication.
• Ability to validate migrations using row counts, checksums, sample queries, and ETL runtime comparisons.
• Experience recreating secret scopes in AWS Secrets Manager.
• Knowledge of AWS S3, Glue, Secrets Manager, IAM basics, Athena, Lake Formation, and EMR.
• Programming experience with Spark SQL, PySpark, Java, Python, and Scala; Agile pod delivery experience including daily ticket updates, standups, and code review.

Culture & Benefits

• Remote home-office work in Colombia.
• Collaboration with teams across more than 25 countries.
• AI tools and automation are integrated into daily engineering workflows.