Data Engineer (AI)

Location: New York, NY, United States; onsite

Base salary: $150,000–$200,000 annually, plus potential discretionary annual incentive and benefits.

Company

An investment firm focused on private markets, bringing together people, ideas, and capital to support companies and investors.

What you will do

• Build, scale, and maintain robust data solutions and pipelines from financial data vendors to gold tables for machine learning use cases.
• Design and optimize extraction, loading, transformation, and orchestration workflows for scalability, reliability, maintainability, and speed.
• Lead end-to-end software development projects involving large language models, retrieval-augmented generation, and other AI technologies.
• Apply CI/CD, infrastructure-as-code, containerization, and cloud-native deployment practices.
• Translate business use cases into production-ready services, owning systems from concept through production.
• Implement testing and monitoring for data quality, integrity, and system reliability while mentoring junior team members.

Requirements

• Bachelor’s degree required; a concentration in computer science, mathematics, physics, STEM, or engineering is preferred.
• At least 6 years of experience in data engineering or a related discipline.
• Expertise in Python and SQL, with experience in data manipulation, analysis, large-scale data projects, data cleaning, ETL, and information retrieval.
• Proficiency with Databricks/PySpark, dbt, workflow orchestration tools such as Airflow or Temporal, and data warehousing and transformation.
• Experience with LLMs, prompt engineering, RAG frameworks, and/or vector databases; excellent verbal and written communication skills.
• Must work onsite in New York, NY and be willing to travel up to 20% to collaborate with distributed team members.

Nice to have

• Experience in financial services or private equity.
• Knowledge of machine learning fundamentals, feature engineering, and knowledge graphs.
• Experience in product development or financial services environments.

Culture & Benefits

• Retirement benefits, health insurance, life insurance, and disability coverage.
• Paid time off and paid holidays.
• Family planning benefits and wellness programs.
• Potential participation in an annual discretionary incentive program based on individual and organizational performance.
• Collaboration with distributed team members across locations.