Python Developer with AI Evaluation Frameworks

Location: Armenia, Bulgaria, Cyprus, Georgia, Kazakhstan, Latvia, Poland, Romania, Serbia, or Ukraine

Company

Global technology projects are delivered in a supportive, flexible, and innovative environment, with opportunities for professional development and IT certification support.

What you will do

• Develop custom AI evaluation services with Python and AWS Lambda.
• Build LLM-as-a-judge evaluators for subjective quality assessment of AI-generated content.
• Implement JSON Schema-based tool response validation, workflow contract checks, numerical accuracy validation, and trace-level evaluation controls.
• Develop PII detection using pattern matching and AWS Bedrock Guardrails.
• Integrate evaluation services with enterprise AI platforms and workflows.
• Document evaluation rules and quality metrics while supporting monitoring, troubleshooting, and continuous improvement.

Requirements

• 3+ years of experience in Python software engineering.
• Experience building quality assurance, validation, or evaluation frameworks for AI or machine learning systems.
• Experience developing and deploying AWS Lambda functions and backend services.
• Knowledge of prompt engineering, AI quality metrics, testing methodologies, and governance concepts.
• Experience with JSON-based APIs, schema validation, cloud environments, and modern software development practices.
• Strong analytical and problem-solving skills.

Nice to have

• Experience registering custom evaluators in AWS AgentCore Evaluation.
• Experience with AWS Bedrock Guardrails, CloudWatch Logs, and AI observability frameworks.
• Knowledge of enterprise AI governance, compliance requirements, and LLM benchmarking.
• Experience building automated quality-control solutions for enterprise-scale AI workloads.

Culture & Benefits

• Work on global projects within a cross-functional team of AI engineers, backend developers, platform specialists, architects, and governance stakeholders.
• Vacation according to the laws of the employee’s country.
• Health insurance support for employees and their loved ones.
• 10 sick days without a doctor's note, followed by the applicable country laws.
• Paid time off for state holidays, plus certification and learning support.