Senior PySpark Developer (PySpark)

Location: Porto or Lisbon, Portugal (Hybrid)

Salary: €28,000–€52,000 gross per year

Company

hirify.global is a Portugal-based IT consulting partner with 18 years of experience, delivering technology consulting and software projects.

What you will do

• Develop scalable batch and streaming data pipelines with PySpark and Apache Spark.
• Integrate data from SQL and NoSQL databases, APIs, files, and Kafka streams.
• Optimize Spark workloads through partitioning, shuffling, caching, memory tuning, and data-skew management.
• Work with Parquet, Delta Lake, Avro, cloud platforms, and data-processing environments such as Databricks or EMR.
• Build pipeline orchestration and delivery workflows using Airflow, Prefect, Git, and CI/CD.
• Collaborate in Agile environments and contribute to testing, performance analysis, and problem solving.

Requirements

• At least 5 years of experience in software development or data engineering.
• Strong professional experience with PySpark and Apache Spark.
• Advanced proficiency in Python and SQL.
• Deep understanding of Spark architecture, DAGs, lazy evaluation, caching, partitioning, shuffling, memory tuning, and data skew.
• Experience with cloud platforms such as AWS, Azure, or GCP, plus Databricks, EMR, or similar platforms.
• Experience with PyTest, unit and integration testing, Spark data mocking, Git, and CI/CD pipelines.

Culture & Benefits

• No-term full-time contract.
• Health insurance and 22 days of paid vacation.
• Four additional annual days for Carnival, Christmas Eve, New Year's Eve, and the employee's birthday.
• Meal allowance through Coverflex, referral bonus, discounts, and flexible options.
• Annual training budget and challenging projects with opportunities to grow and learn.