1 день назад
Data Engineer (AWS/Spark)
Вакансия из Hirify Global, списка международных tech-компанийДля мэтча и отклика нужен Plus
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Data Engineer (AWS/Spark): Building and operating production data platform components, ETL pipelines, ingestion flows, and analytical data models with an accent on Spark on EMR, Airflow orchestration, data quality, and OLAP performance. Focus on designing incremental and idempotent pipelines, handling schema changes, tuning infrastructure costs, and delivering reliable datasets for analysts and business stakeholders.
Location: Irvine, California, United States; on-site
Company
develops networking devices and smart home products for customers in more than 170 countries.
What you will do
- Own data platform components end to end, including pipelines, tables, quality checks, monitoring, and recovery.
- Design and build ETL on EMR with Spark and Airflow, including OLTP ingestion through DataX or CDC tools.
- Develop dimensional data models and ODS, DWD, DWS, and ADS layers for analytical use cases.
- Implement data quality checks, freshness monitoring, issue resolution, and reusable engineering components.
- Tune job performance and infrastructure costs across EMR and StarRocks.
- Partner with analysts and business stakeholders to turn requirements into usable datasets.
Requirements
- Bachelor’s degree in Computer Science, Information Systems, or a related field, or equivalent practical experience.
- At least 2 years of hands-on data development in a production environment and ownership of production pipelines.
- Strong SQL and production-grade Python skills, including PySpark, pandas, boto3, testing, error handling, and logging.
- Production experience with Spark on EMR or equivalent Hadoop/Hive, Airflow, and a data ingestion or CDC tool.
- Experience with StarRocks, Doris, or ClickHouse; dimensional modeling; slowly changing dimensions; AWS; Linux; and Git.
- Ability to use AI effectively for SQL, debugging, and unfamiliar schemas while validating its output.
Nice to have
- AWS cost optimization, including EMR sizing, Spot strategy, S3 lifecycle policies, and StarRocks versus Athena trade-offs.
- Experience with Iceberg, Hudi, Delta, Kafka, Flink, Spark Structured Streaming, dbt, Glue Data Catalog, data lineage, or data quality frameworks.
- Experience designing QuickSight datasets, SPICE usage, and row-level permissions.
Culture & Benefits
- Fully paid medical, dental, and vision insurance, with partial dependent coverage.
- 401(k) contributions, bi-annual reviews, and annual pay increases.
- Health and wellness benefits, including a free gym membership.
- Free snacks and drinks and quarterly team-building events.
- Visa sponsorship is not available.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →