Senior Backend Software Engineer (AI)

TL;DR

Senior Backend Software Engineer (AI): Developing strategies and frameworks to capture web data at scale using agentic tooling and AI with an accent on high-volume data extraction and pipeline optimization. Focus on building robust ETL/ELT pipelines using Apache Spark and implementing adaptive crawling strategies to unlock medical insights.

Location: Hybrid in New York

Company

hirify.global provides a platform that harnesses data and AI technology to provide optimal healthcare information for doctors globally to improve patient outcomes.

What you will do

• Develop strategies and frameworks to capture web data at scale using agentic tooling.
• Design and maintain scalable data extraction frameworks for structured and unstructured data.
• Build and optimize robust ETL/ELT pipelines using Apache Spark on AWS EMR.
• Collaborate with senior engineers to evolve data architecture and platform scalability.
• Lead data integration efforts across multiple systems to ensure accuracy.
• Monitor and troubleshoot data flows to resolve performance issues proactively.

Requirements

• Proficiency in web scraping strategies (curl, network analysis, proxies, selenium/playwright).
• Experience building agentic web scraping solutions leveraging AI for autonomous decision-making.
• Strong SQL skills and professional experience with PostgreSQL.
• Experience with big data tools like Apache Spark, preferably on AWS EMR.
• Experience with Docker or other containerization technologies.
• Must be based in New York (Hybrid)

Nice to have

• Understanding of Large Language Models (LLMs) and their applications.
• Familiarity with model training and fine-tuning, particularly in NLP contexts.
• Knowledge of network, security, and encryption protocols (HTTP/HTTPS/TLS).

Culture & Benefits

• Commitment to diversity, equity, and an inclusive workplace.
• Equal opportunity employer policies.
• Provision of reasonable accommodations for applicants with disabilities.