Senior Software Engineer (Web Scraping)

TL;DR

Senior Software Engineer (Python/Web Scraping): Designing and building distributed scraping architectures to collect data from thousands of sources with an accent on bypassing anti-bot measures and reverse engineering APIs. Focus on scaling data acquisition, implementing anomaly detection, and optimizing high-volume data extraction.

Location: Lisbon, Portugal

Compensation: EUR 46,000 - 55,000 yearly

Company

NIQ is the world’s leading consumer intelligence company, delivering comprehensive understanding of consumer buying behavior through advanced analytics.

What you will do

• Design and build distributed scraping architectures scalable across thousands of domains.
• Reverse engineer APIs and mobile apps using tools like Frida, mitmproxy, Charles Proxy, and Burp.
• Detect and bypass anti-bot measures, including JavaScript checks and browser behavior detection.
• Lead technical innovation by prototyping new scraping approaches and identifying edge cases.
• Set up anomaly detection and monitoring to detect blockers or failures early.

Requirements

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies, WAFs, fingerprinting, and JavaScript challenges.
• Experience with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers.
• Expertise in handling residential/datacenter proxies, session management, and TLS tweaking.
• Strong debugging skills across browser, HTTP traffic, and device-level interactions.
• Knowledge of MongoDB, InfluxDB, and Redis.

Nice to have

• Experience with real-device simulation for mobile scraping.
• Proficiency in building distributed task queues using Celery or Kafka.
• Experience with AWS, GCP, Kubernetes, Cloud Run, or Cloud Functions.
• Experience scraping high-security platforms or mobile-first environments.

Culture & Benefits

• Ownership and freedom to experiment and shape the company's scraping strategy.
• Flexible working environment and flexible hours.
• Access to LinkedIn Learning and Employee-Assistance-Program (EAP).
• Volunteer time off.
• Collaborative environment within a global team of 60+ data experts.