Technical Lead, AI Benchmarking and Evaluation Research (Remote, ROU) (AI)

Location: Romania - Remote

Company

CrowdStrike is a global cybersecurity company developing an AI-native platform for protecting organizations and processing large-scale security data.

What you will do

• Design and build benchmark datasets for cybersecurity agent tasks, including expert-labeled ground truth, scoring rubrics, and releases.
• Build and calibrate LLM-based and programmatic judges against expert labels, measuring agreement and documenting disagreements.
• Develop reproducible evaluation harnesses that capture agent actions and traces, not only transcripts.
• Run scorecards for CrowdStrike agents and frontier models, reporting protection, correctness, and usefulness.
• Investigate agent failure modes and publish internal reports, papers, and talks.
• Lead and grow researchers and engineers while setting the evaluation strategy, methodology, roadmap, and success metrics.

Requirements

• Hands-on cybersecurity expertise in malware analysis, reverse engineering, incident response, threat hunting, detection engineering, or SOC operations.
• Experience building evaluations, benchmarks, or datasets for AI or LLM systems with shareable artifacts.
• Strong Python and data tooling skills, including production-quality evaluation code, harnesses, and analysis.
• Practical experience with LLMs and agentic systems, including tool use, multi-step planning, sandboxing, instrumentation, and tracing.
• At least 2 years of technical leadership experience as a team lead, tech lead, or manager while remaining hands-on.
• Ability to standardize reproducible testing pipelines, communicate measured findings, and explain limitations to technical and executive audiences.

Nice to have

• Published research or talks on AI evaluation, LLM judging, or AI security.
• Experience with adversarial testing, AI red teaming, SOC platforms, SIEM/SOAR tooling, or detection engineering.
• Knowledge of MITRE ATT&CK and security certifications such as GREM, GCFA, GCIH, GCIA, or OSCP.
• Experience fine-tuning or distilling models on expert-labeled decision data.

Culture & Benefits

• Remote work with flexibility and autonomy.
• Market-leading compensation and equity awards.
• Physical and mental wellness programs, competitive vacation, and holidays.
• Paid parental and adoption leave.
• Professional development, employee networks, neighborhood groups, and volunteer opportunities.