Staff / Senior Software Engineer (Agentic Search) - Crawler

Location: Zurich, Switzerland

Company

Nebius builds a full-stack AI cloud platform and an agent-native search platform for programmatic retrieval of fresh, verified web data.

What you will do

• Design, implement, and operate web-scale crawling systems for internet content acquisition.
• Build ingestion workflows for crawlers, structured feeds, and partner integrations.
• Develop crawl scheduling, prioritization, recrawl policies, freshness strategies, URL discovery, deduplication, and content extraction.
• Operate reliable, high-throughput infrastructure processing billions of URLs and monitor resource, bandwidth, and infrastructure costs.
• Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality.
• Collaborate with indexing and ML teams and enable safe experimentation with crawling strategies.

Requirements

• 5+ years of experience building backend or distributed systems.
• Strong Go or C++ expertise.
• Experience with large-scale distributed systems, including 10k+ RPS, billions of URLs, or high-throughput pipelines.
• Understanding of HTTP, DNS, TLS, crawling, scraping, and content extraction.
• Experience operating production systems and debugging failures in distributed environments.
• Strong understanding of scalability, fault tolerance, and resource management.

Nice to have

• Experience with web crawling, streaming data pipelines, event-driven systems, distributed schedulers, queues, or asynchronous processing.
• Knowledge of Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, or MapReduce.
• Experience in ad tech, social networks, search engines, or other large-scale content platforms.

Culture & Benefits

• Competitive compensation and career growth opportunities.
• Learning opportunities, flexibility, and ownership.
• Collaborative, innovative, and international environment.
• Opportunity to work on impactful AI projects with experienced engineering teams.
• Applicants must be authorized to work in Switzerland and provide proof of employment eligibility.

Hiring process

• Coding interviews are part of the selection process.