Senior Web Scraper (Python)

TL;DR

Senior Web Scraper (Python): Designing and optimizing robust data collection systems using Scrapy and Python with an accent on bypassing anti-bot countermeasures and ensuring data resilience. Focus on architecting scalable scraping solutions, implementing complex crawling logic, and maintaining high standards of code quality.

Location: Kuala Lumpur, Malaysia

Company

Global consumer intelligence company providing real-time market visibility and retail analytics through advanced data collection and processing.

What you will do

• Design, implement, and document robust Scrapy spiders to ensure resilience against website changes.
• Develop sophisticated web crawling solutions to bypass advanced anti-bot countermeasures.
• Conduct comprehensive code reviews and data validation to maintain high standards of quality.
• Architect significant new developments across multiple codebases and provide technical documentation.
• Provide training sessions for relevant teams on new scraping implementations.

Requirements

• Master's degree in Computer Science, IT, or a related field.
• 5-6 years of professional experience in software engineering.
• Proficiency with BeautifulSoup or Scrapy framework.
• Solid understanding of HTML, JavaScript, and single-page applications (SPA).
• Experience with RESTful/GraphQL APIs and Python web frameworks like Django or FastAPI.
• Strong skills in Docker, Git, pandas, regular expressions, Linux, and bash scripting.

Nice to have

• Experience with time series databases, such as InfluxDB.
• Hands-on experience with major cloud providers such as AWS, GCP, or Azure.

Culture & Benefits

• Flexible working environment.
• Volunteer time off.
• Access to LinkedIn Learning.
• Employee-Assistance-Program (EAP).