Software Engineer (Web Crawling)

TL;DR

Software Engineer (Web Crawling): Develop and optimize web crawlers and scraping infrastructure with an accent on handling anti-bot mechanisms, performance, and security challenges. Focus on designing scalable crawling operations and integrating AI solutions to enhance automation and data extraction accuracy.

What you will do

• Develop, enhance, and maintain web crawlers and scraping infrastructure.
• Optimize scraping techniques to handle anti-bot mechanisms, performance, and security challenges.
• Collaborate with a geographically distributed team to identify and resolve issues.
• Ensure high availability, efficiency, and reliability of crawling operations.
• Integrate AI solutions to enhance automation and data extraction accuracy.

Requirements

• 3+ years of experience in software engineering with a focus on web crawling and data extraction.
• Strong expertise in Node.js for web crawling applications.
• Deep understanding of HTML, JavaScript, and reverse engineering techniques.
• Hands-on experience with Playwright, Puppeteer, and Cheerio for automation and scraping.
• Knowledge of security and performance best practices related to web crawling.

Nice to have

• Experience with Apify or Crawlee for large-scale crawling solutions.
• Proficiency in TypeScript.

Culture & Benefits

• Unlimited PTO.
• Comprehensive medical, dental, and vision insurance plans.
• STD, LTD, AD&D, and life insurance coverage.
• Free membership to TalkSpace, Teladoc and Health Advocate.
• 401(k) retirement plan with company matching.
• Pre-tax commuter benefits and free equipment including laptop and home office stipend.