Principal Software Engineer (AI Platform Engineering)

TL;DR

Principal Software Engineer (AI Platform Engineering): Designing and governing the architectural direction of training data flows across the AI Platform with an accent on tenant isolation, PII-free signals, and end-to-end traceability. Focus on building TB-scale pipelines, managing high-performance vector databases, and implementing scalable RAG data pipelines.

Company

hirify.global is a leader in identity security, providing an AI-powered platform for governing and securing access to applications and data for global enterprises.

What you will do

• Architect and operate the AI Data Lake on GCS, including bucket layout, tiered separation, and encryption.
• Develop TB-scale batch and streaming pipelines using Spark on Dataproc and Apache Beam on Dataflow.
• Implement orchestration via Flyte and manage schema evolution using Avro and Protobuf.
• Build and maintain feature stores (Feast) and vector databases (Qdrant, Pgvector) for production-scale embedding storage.
• Develop RAG data pipelines for chunking, encoding, and upserting document embeddings.
• Design data anonymization and labeling microservices to ensure strict multi-tenant isolation and PII removal.

Requirements

• 8+ years of production-scale data engineering experience.
• Proven track record of designing and operating an end-to-end production data lake.
• Deep expertise in PySpark/Scala, Apache Beam, and Dataflow.
• Experience with orchestration tools such as Flyte, Kubeflow, or Airflow.
• Practical experience with vector databases (Qdrant, Pgvector) and RAG fundamentals.
• Bachelor's degree in Computer Science, Engineering, or equivalent practical/military experience.

Nice to have

• Experience with differential privacy or k-anonymity for ML datasets.
• Open source contributions to Feast, Great Expectations, Apache Beam, or dbt.
• Familiarity with IAM and access governance data (entitlements, access graphs).
• Experience managing Iceberg or Delta Lake at petabyte scale.

Culture & Benefits

• Opportunity to work on a large-scale, Kubernetes-based SaaS platform.
• Engagement with challenging cloud and reliability problems at scale.
• Collaborative, reliability-focused engineering culture.
• Competitive compensation, benefits, and professional growth opportunities.