Senior Python Developer (Web Scraping)

TL;DR

Senior Python Developer (Web Scraping): Designing and implementing robust Scrapy spiders to collect large-scale retail data with an accent on anti-bot bypass and data resilience. Focus on architecting sophisticated crawling solutions, optimizing data validation, and integrating with cloud-based infrastructures.

Location: Kuala Lumpur, Malaysia

Company

NIQ is a leading global consumer intelligence company providing real-time visibility into retail markets through advanced analytics and data collection.

What you will do

• Design, implement, and document robust Scrapy spiders to ensure resilient data collection against website changes.
• Develop sophisticated web crawling solutions to bypass advanced anti-bot countermeasures using knowledge of HTTP and browser mechanisms.
• Conduct comprehensive code reviews and data validation to maintain high standards of code and data quality.
• Architect and contribute to significant new developments across multiple codebases.
• Provide technical documentation and conduct training sessions for relevant internal teams.

Requirements

• Master's degree in Computer Science, IT, or a related field.
• At least 3 years of professional experience in software engineering.
• Proficiency with BeautifulSoup or Scrapy framework.
• Strong understanding of HTML, JavaScript, and single-page applications (SPAs).
• Experience with RESTful/GraphQL APIs and Python frameworks like Django or FastAPI.
• Hands-on experience with Docker, Git, pandas, regular expressions, Linux, and bash scripting.

Nice to have

• Experience with time series databases, such as InfluxDB.
• Proven experience with major cloud providers (AWS, GCP, or Azure).

Culture & Benefits

• Flexible working environment.
• Professional development through LinkedIn Learning.
• Employee Assistance Program (EAP).
• Volunteer time off.