Software Engineers (AI Evaluation)

Location: Remote within Argentina, Brazil, Chile, Colombia, Ecuador, Mexico, Portugal, Spain, or Uruguay

Salary: $65 per hour

Company

hirify.global is recruiting software engineers for a specialized project evaluating real-world software engineering tasks from open-source repositories.

What you will do

• Review coding tasks derived from real GitHub issues, pull requests, bug fixes, and feature implementations.
• Assess whether problem statements, success criteria, tests, dependencies, and repository setup are clear and complete.
• Evaluate unit tests for correctness, coverage, robustness, and appropriate implementation flexibility.
• Identify flaky tests, missing dependencies, version conflicts, environment-specific behavior, and reproducibility issues.
• Assess task difficulty, multi-file and cross-module complexity, and real-world relevance.
• Provide written recommendations on whether tasks should be accepted, improved, or excluded.

Requirements

• 3+ years of professional software engineering experience.
• Strong experience with large, multi-file codebases, production-code maintenance, debugging, and pull request review.
• Strong understanding of unit testing, test coverage, Git, and GitHub-based development workflows.
• Experience with dependency management, environment setup, reproducibility, and evaluating technical problems.
• Ability to provide clear written technical feedback.

Nice to have

• Contributions to or maintenance of open-source projects.
• Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks.
• Experience with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest.
• Experience with Docker, CI/CD, pip, conda, dependency pinning, test fixtures, test isolation, or property-based testing.
• Experience with technical assessments, AI/ML evaluation, data curation, RLHF, or benchmark development.

Culture & Benefits

• Remote, part-time, project-based consulting engagement.
• Focus on software engineering, open-source code review, testing, and technical evaluation.
• Work on real-world repositories and professional software engineering problems.