TL;DR

Data Engineer (Azure): Designing, developing, and maintaining real-time data streaming pipelines for machine logs with an accent on efficient data flow, transformation, and integration into data lakes. Focus on implementing optimized data processing solutions using Python and applying software engineering best practices.

Location: Remote from Vietnam for a US client

Company

%hirify_global% is a global cloud computing company providing public cloud offerings and delivery services, specifically in cloud data services.

What you will do

Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions.
Load, merge, and process machine logs from Kafka.
Integrate processed data into Redis cache and send it to the data lake.
Implement and optimize data processing solutions using Python.
Apply software engineering best practices, including code reviews and CI/CD.
Collaborate with cross-functional teams to deliver high-quality data solutions.

Requirements

Proven experience in real-time data streaming using Spark Streaming or Azure Functions.
Strong proficiency in Python for data processing and automation tasks.
Hands-on experience with Kafka for data ingestion and message queuing.
Familiarity with Redis for caching and fast data retrieval.
Knowledge of data lake architectures and best practices.
Solid understanding of software engineering principles.
Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.

Nice to have

Experience with cloud platforms such as Azure.
Knowledge of other programming languages and frameworks related to data engineering.
Familiarity with data modeling, ETL processes, and data lake concepts.