Web Crawler Engineer (C++/Rust)

Location: On-site in San Francisco, California; the logistics section specifies an in-person opportunity in Zurich.

Salary: $180K–$350K per year, plus equity.

Company

hirify.global is an applied AI lab building a large-scale search engine, web-crawling infrastructure, embedding models, and vector databases.

What you will do

• Build a distributed crawler capable of handling 100M+ pages per day.
• Optimize crawl politeness and rate limiting across thousands of domains.
• Design systems for dynamic content, JavaScript rendering, and anti-bot measures.
• Create intelligent crawl scheduling and prioritization algorithms to maximize coverage efficiency.

Requirements

• Extensive experience building and scaling web crawlers, or the ability to ramp up quickly.
• Experience with a high-performance language such as C++ or Rust.
• Familiarity with TypeScript, Playwright, modern web design, and the Chrome DevTools Protocol.
• Comfort optimizing systems to an exceptional degree.
• Interest in finding high-quality knowledge and understanding its importance.

Culture & Benefits

• In-person work is required.
• Visa sponsorship is available for international candidates, including STEM OPT, OPT, H1B, O1, and E3; sponsorship is not guaranteed.
• Premium medical, dental, and vision benefits.
• Fertility benefits and 16 weeks of fully paid parental leave for all new parents.
• Monthly wellness stipend.