Web Scraper (Python)

TL;DR

Web Scraper (Python): Developing and optimizing robust web crawling solutions for retail analytics with an accent on bypassing anti-bot countermeasures and maintaining high-scale data collection. Focus on architecting Scrapy spiders, integrating time-series databases, and ensuring data quality across multiple codebases.

Location: Kuala Lumpur, Malaysia

Company

hirify.global is the world’s leading consumer intelligence company, delivering comprehensive insights into consumer buying behavior and retail analytics.

What you will do

• Design, implement, and document robust Scrapy spiders ensuring resilience against website changes.
• Develop sophisticated web crawling solutions to bypass advanced anti-bot countermeasures using HTTP protocol and browser mechanisms.
• Conduct comprehensive code reviews and data validation to maintain high standards of code and data quality.
• Architect significant new developments across multiple codebases and provide clear technical documentation.
• Conduct training sessions for relevant teams to facilitate knowledge transfer.

Requirements

• Master's degree in Computer Science, IT, or a related field.
• At least 3 years of professional experience in software engineering.
• Proficiency with BeautifulSoup or Scrapy framework, HTML, and JavaScript.
• Solid understanding of single-page applications and experience with RESTful and/or GraphQL APIs.
• Hands-on experience with Python web frameworks such as Django or FastAPI.
• Strong skills in Docker, Git, pandas, regular expressions, Linux, and bash scripting.
• Proven experience with major cloud providers such as AWS, GCP, or Azure.

Nice to have

• Experience with time series databases, such as InfluxDB.

Culture & Benefits

• Flexible working environment.
• Volunteer time off.
• Access to LinkedIn Learning.
• Employee-Assistance-Program (EAP).