Data Engineer (Azure)

TL;DR

Data Engineer (Azure): Designing, developing, and maintaining real-time data streaming pipelines for machine logs with an accent on efficient data flow, transformation, and integration into data lakes. Focus on implementing optimized data processing solutions using Python and applying software engineering best practices.

Location: Remote from Vietnam for a US client

Company

hirify.global is a global cloud computing company providing public cloud offerings and delivery services, specifically in cloud data services.

What you will do

• Design, develop, and maintain real-time data streaming pipelines using Spark Streaming or Azure Functions.
• Load, merge, and process machine logs from Kafka.
• Integrate processed data into Redis cache and send it to the data lake.
• Implement and optimize data processing solutions using Python.
• Apply software engineering best practices, including code reviews and CI/CD.
• Collaborate with cross-functional teams to deliver high-quality data solutions.

Requirements

• Proven experience in real-time data streaming using Spark Streaming or Azure Functions.
• Strong proficiency in Python for data processing and automation tasks.
• Hands-on experience with Kafka for data ingestion and message queuing.
• Familiarity with Redis for caching and fast data retrieval.
• Knowledge of data lake architectures and best practices.
• Solid understanding of software engineering principles.
• Strong problem-solving skills and ability to work in a fast-paced, collaborative environment.

Nice to have

• Experience with cloud platforms such as Azure.
• Knowledge of other programming languages and frameworks related to data engineering.
• Familiarity with data modeling, ETL processes, and data lake concepts.