13 дней назад
Benchmark & Evaluation Engineer (AI)
Вакансия из Hirify Global, списка международных tech-компанийДля мэтча и отклика нужен Plus
Мэтч & Сопровод
Для мэтча с этой вакансией нужен Plus
Описание вакансии
Текст:
TL;DR
Benchmark & Evaluation Engineer (AI): Building production-scale enterprise data environments, benchmark datasets, and evaluation harnesses for AI analysts with an accent on advanced SQL, data modeling, synthetic data, and rigorous scoring. Focus on designing multi-system data environments, calibrating golden SQL and LLM judges, preventing data leakage, and tracking evaluation regressions.
Location: Remote within the Philippines; US Shift schedule.
Company
is a tech-enabled workforce augmentation platform connecting businesses with offshore talent and technology. The client develops AI platforms and industry-standard benchmarks for evaluating AI analysts working with complex enterprise data.
What you will do
- Design and deploy large-scale, multi-system enterprise data environments across warehouses, data lakes, operational databases, and APIs.
- Generate calibrated benchmark datasets with realistic noise, seasonality, data drift, and synthetic PII.
- Create and verify benchmark tasks, golden SQL, and rubric-based reasoning keys while preventing data leakage and ambiguity.
- Extend evaluation harnesses with deterministic checks and LLM judges.
- Maintain regression-tracked leaderboards and ensure scoring integrity.
Requirements
- Expert production-grade SQL skills, including CTEs, window functions, query plans, and cardinality analysis.
- Strong data modeling experience with star and snowflake schemas, SCDs, and referential integrity.
- Strong Python skills with pandas, NumPy, and Polars.
- Hands-on experience loading data into at least one major warehouse or database: Snowflake, BigQuery, Redshift, or Postgres.
- Experience designing evaluation sets or benchmark tasks, verifying golden answers and SQL, calibrating rubrics or LLM judges, preventing data leakage, and tracking regressions.
- Ability to construct realistic multi-system environments with large-scale, semi-structured, or dirty synthetic data.
Nice to have
- Domain knowledge in healthcare or health insurance, finance or FP&A, product analytics, or supply chain.
- Experience with LLM evaluation frameworks such as SWE-bench or Cursor-style harnesses.
- Experience with streaming or data-lake technologies such as S3, Parquet, dbt, Airflow, Spark, or DuckDB.
Culture & Benefits
- Full-time remote work from the Philippines.
- US Shift schedule.
- Work with a remote-first, fast-growing AI platform environment.
- Contribute to products spanning natural language analytics, data modeling, and end-to-end data systems.
Будьте осторожны: если работодатель просит войти в их систему, используя iCloud/Google, прислать код/пароль, запустить код/ПО, не делайте этого - это мошенники. Обязательно жмите "Пожаловаться" или пишите в поддержку. Подробнее в гайде →
Музыка
Осень
Поддержка