Senior Software Engineer (Agentic Search) - Crawler (AI)

TL;DR

Senior Software Engineer (Crawler) (AI): Building a web-scale content acquisition and crawling infrastructure for an agent-native search platform with an accent on distributed systems and high-throughput data flows. Focus on designing URL discovery, deduplication, and recrawl policies to ensure fresh, verified data for AI agents.

Location: Amsterdam, Netherlands or London, United Kingdom. Applicants must be authorized to work in the country in which they apply.

Company

Nebius is building a full-stack AI cloud platform for developers and enterprises, providing GPU orchestration and inference optimization.

What you will do

• Design and operate web-scale crawling systems for content acquisition from the open web.
• Build ingestion workflows for internal and external data sources, including structured feeds and partner integrations.
• Develop crawl scheduling, prioritization, and freshness strategies for internet-scale data.
• Implement systems for URL discovery, deduplication, and content extraction.
• Ensure infrastructure reliability and observability under high-throughput conditions.
• Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements.

Requirements

• 5+ years of experience building backend or distributed systems.
• Strong expertise in Go or C++.
• Experience with large-scale systems handling 10k+ RPS and billions of URLs.
• Deep understanding of web protocols (HTTP, DNS, TLS), crawling, and scraping.
• Proven ability to operate and debug production systems in distributed environments.
• Must have legal authorization to work in the Netherlands or the United Kingdom.

Nice to have

• Experience with streaming data pipelines and messaging platforms (Kafka, Pulsar, NATS, RabbitMQ).
• Knowledge of designing distributed schedulers and asynchronous processing systems.
• Experience with Spark, Flink, Beam, or MapReduce.
• Background in ad tech, social networks, search engines, or large-scale content platforms.

Culture & Benefits

• Competitive compensation and career growth opportunities.
• International environment with highly talented teams.
• Culture of flexibility, ownership, and bold thinking.
• Opportunity to work on impactful, cutting-edge AI projects.

Hiring process

• Coding interviews are conducted as part of the selection process.