Research Engineer, Benchmarks & Evaluation (AI)

Location: Hybrid in New York, NY or San Francisco, CA, United States

Salary: $204.4K–$352K annually, plus equity

Company

Ramp builds financial infrastructure that automates payment authorization, risk flagging, spend categorization, and accounting workflows for businesses. Ramp Labs is the company’s AI research group, developing finance-focused models, environments, evaluations, and products.

What you will do

• Build versioned benchmark suites from agent traces, customer edge cases, expert labels, and real finance workflows such as expense policy review, invoice coding, month-end close, and spreadsheet Q&A.
• Run benchmark suites across models and agent versions, decide what to measure, and report results used for engineering decisions.
• Design rubrics, programmatic checks, and LLM-as-judge graders calibrated against human agreement.
• Convert evaluations into reward functions and RL environments while detecting reward hacking and contamination.
• Publish external benchmarks and technical write-ups on model performance in real finance work.

Requirements

• Hands-on experience building LLM benchmarks or evaluations used to make real decisions.
• Strong Python and software engineering skills; evaluation code must be reproducible and fast.
• Statistical judgment covering variance, confidence intervals, sample size, inter-rater agreement, and distinguishing meaningful deltas from noise.
• Working knowledge of post-training, including SFT and reinforcement learning with verifiable rewards, and how evaluations feed into it.
• Comfort with SQL and large datasets, including working directly from production traces.
• Strong judgment about which measurements are valuable.

Nice to have

• Industry experience on an evaluation, benchmarking, or post-training team.
• Experience building RL environments or verifiers for agentic tasks.
• Published benchmark or evaluation research, including at NeurIPS Datasets & Benchmarks, ICML, or ACL.
• Experience building or contributing to evaluation frameworks such as verifiers, lm-evaluation-harness, or HELM.
• Familiarity with accounting, finance, or other expert-domain work.
• Public work such as a benchmark, evaluation suite, or failure-analysis write-up.

Culture & Benefits

• Hybrid work in New York or San Francisco with relocation expense coverage to NYC or SF if needed.
• Full-time benefits include flexible PTO, home-office equipment ordering, a health and wellness stipend, intra-office travel budget, and weekly coffee stipend.
• United States benefits include medical, dental, and vision coverage, One Medical membership, 401(k) with employer match, parental leave, fertility HRA, pet insurance, and in-office meals and refreshments.
• Ramp considers qualified applicants with arrest and conviction records under the San Francisco Fair Chance Ordinance.

Hiring process

• Referred candidates must contact the referring person so that person can apply on their behalf.
• Ramp will contact candidates only through official Ramp.com email addresses and will not request payment or sensitive personal information.