Data Engineer (AI)

Location: In-person in San Francisco, California, United States

Salary: $180K–$350K per year plus equity

Company

hirify.global is an applied AI lab building large-scale web search infrastructure, embedding models, and vector databases.

What you will do

• Design a lakehouse architecture for more than 100 PB of web crawl data.
• Build streaming pipelines processing billions of documents per day for real-time indexing.
• Architect the data layer supporting embedding model training on Ray.
• Scale ClickHouse deployments for analytical queries across petabytes of search logs.
• Build and operate data infrastructure spanning web crawling, model training, and real-time search.

Requirements

• Deep understanding of lakehouse architectures, including Delta Lake, Iceberg, or Hudi.
• Experience building and operating large-scale distributed data processing pipelines.
• Hands-on experience with streaming systems such as Kafka or Flink.
• Production-scale familiarity with Ray, Spark, or ClickHouse.
• Strong focus on reliability and robust system operation.
• Ability to work in person in San Francisco.

Nice to have

• Experience with Lance or other vector-native storage formats.
• Background in GPU-accelerated data processing with RAPIDS or cuDF.

Culture & Benefits

• Visa sponsorship is available for international candidates, including STEM OPT, OPT, H1B, O1, and E3 visas.
• Premium medical, dental, and vision healthcare benefits.
• Fertility benefits.
• 16 weeks of fully paid parental leave for all new parents.
• Monthly wellness stipend.