Backend Software Engineer (Search)

TL;DR

Backend Software Engineer (Search): Design, develop, and operate large-scale web crawlers, ingestion pipelines, and data processing systems with an accent on backend and frontend components of data acquisition. Focus on building scalable distributed systems, optimizing high-load services, and collaborating across teams to manage web-scale data.

What you will do

• Lead projects developing large-scale web crawlers and ingestion pipelines.
• Build and optimize backend and frontend components for crawler services including data storage and UI dashboards.
• Collaborate with Search and Infrastructure teams to ensure reliable ingestion and processing of billions of web pages.
• Architect and implement fullstack features and scalable distributed systems for real-time data operations.
• Continuously enhance system performance, usability, and reliability through rapid iteration and experimentation.

Requirements

• Minimum 5 years of software development experience with strong knowledge of data structures and algorithms.
• Proficiency in at least one of Python, C++, Rust, or Go.
• Experience with large-scale web crawlers is highly desirable.
• Proven experience building and optimizing high-load, distributed, and hardware-adjacent services.
• Hands-on experience with cloud infrastructure, Kubernetes, and AWS.
• English proficiency at least B2 level.