Associate Performance Engineer (AWS/PySpark)

TL;DR

Associate Performance Engineer (AWS/PySpark): Tuning and troubleshooting cloud-based enterprise data platforms for a federal financial regulatory agency with an accent on Spark/PySpark performance and query optimization. Focus on optimizing data pipelines, resolving production bottlenecks, and ensuring reliability for terabyte-to-petabyte data volumes.

Location: Hybrid in Columbia, MD. Must reside in the United States to obtain a U.S. Federal Public Trust security clearance.

Salary: $93,000 - $136,400

Company

A modern digital services company partnering with government agencies to improve public well-being through intuitive products and services.

What you will do

• Analyze and tune Spark/PySpark and AWS Glue job performance, optimizing partitioning, clustering, and file formats (Parquet, ORC, Iceberg).
• Optimize query performance across Athena, Trino, PostgreSQL, and Redshift to ensure efficient report and dashboard execution.
• Diagnose and resolve production bottlenecks and failed jobs to maintain data availability and zero data-loss requirements.
• Conduct periodic load and performance tests, defining benchmarks and drafting recommendations for system improvements.
• Monitor pipeline health using CloudWatch, ETL metadata, and log analysis to optimize error handling.
• Collaborate with developers and stakeholders in an agile environment and participate in production on-call rotations.

Requirements

• Bachelor's degree in CS/Engineering or equivalent specialized experience.
• Minimum 4 years of experience in performance, data, or quality engineering for large-scale, data-centric systems.
• Hands-on experience tuning Apache Spark/PySpark workloads and AWS Glue jobs.
• Proficiency in Python, SQL, and workflow orchestration using Airflow/MWAA.
• Experience optimizing S3-based data lakes and relational databases (PostgreSQL, Redshift, or Oracle).
• Ability to obtain and maintain a Moderate Risk Public Trust clearance; must reside in the United States.

Nice to have

• Experience with AWS cloud cost optimization.
• Familiarity with CI/CD, Infrastructure-as-Code, and federal security compliance (FISMA, NIST 800-53).
• Experience using AI-assisted development tools for debugging and performance tuning.
• Experience working with financial or regulatory datasets.

Culture & Benefits

• Hybrid work environment operating within the Eastern time zone.
• Comprehensive health coverage including medical, dental, and vision.
• 401(k) retirement benefits and paid time off.
• Life and disability insurance.
• Wellness and employee support programs.