Senior Evaluation Algorithm Engineer (AI)

Location: APAC, Australia, Hong Kong, or Taiwan; remote-first, work-from-home arrangement, which may vary depending on the role and team

Company

Binance operates a global blockchain ecosystem offering cryptocurrency trading, finance, payments, education, research, and institutional digital-asset services.

What you will do

• Design end-to-end LLM evaluation plans for dialogue, financial trading, and other business scenarios.
• Build metric systems, evaluation rubrics, datasets, benchmarks, annotation guidelines, and quality-control processes.
• Analyze model capabilities and failure modes, then provide actionable recommendations for model improvement.
• Automate and scale evaluation workflows by developing sustainable evaluation platforms and toolchains.
• Collaborate with algorithm, product, and data teams to translate evaluation findings into R&D directions.

Requirements

• Master’s degree or higher in Computer Science, Artificial Intelligence, Mathematics, Statistics, or a related field.
• Strong algorithmic foundation and understanding of LLM principles, training, and fine-tuning.
• Hands-on LLM evaluation experience at a large technology company, including commercial deployment evaluation.
• Experience with human evaluation, model-based automatic evaluation, LLM-as-a-judge methods, and metric computation.
• Proficiency in Python, evaluation workflow automation, benchmark construction, data processing, evaluation scripting, and result analysis.
• Strong business understanding, communication skills, and experience driving cross-functional collaboration.

Nice to have

• Experience evaluating dialogue systems, AI Agents, or financial and trading LLMs.
• Experience building AI training or evaluation data and annotation systems.
• Familiarity with RLHF, reward models, or preference data.

Culture & Benefits

• Remote-first, work-from-home arrangement, subject to the role and team.
• Collaboration with global talent in a user-centric organization with a flat structure.
• Fast-paced, innovative environment with autonomy and challenging projects.
• Career growth and continuous learning opportunities.
• Competitive salary and company benefits.