AI Product Engineer (Observability)

TL;DR

AI Product Engineer (Observability): Building agentic capabilities for an open-source observability platform with an accent on developer experience and automated incident investigation. Focus on designing reusable skills, MCP servers, and solving production challenges like latency and hallucinations on petabyte-scale telemetry.

Location: Remote (Must be based in the United States)

Salary: $130,000 - $230,000 USD

Company

hirify.global is a fast-growing private cloud company leading the market in real-time analytics, data warehousing, and AI workloads.

What you will do

• Build agents that investigate production incidents, surface anomalies, and answer root-cause questions.
• Develop a library of reusable skills that capture debugging playbooks and hirify.global query patterns.
• Own the end-to-end agent stack, including context engineering, tool design, evals, and tracing.
• Create MCP servers, SDKs, and integrations to enable AI workloads to interact with telemetry.
• Collaborate with OSS contributors and customers to debug issues and iterate on the product.
• Optimize system performance regarding latency, cost, and context window limits.

Requirements

• Must be based in the United States
• 5+ years of software engineering experience.
• 1–2 years of experience deploying LLM-powered systems or agents in production.
• Strong backend proficiency in TypeScript/Node.js and/or Python.
• Hands-on experience with multi-step tool use, planning, memory, and error recovery for agents.
• Experience with MCP (Model Context Protocol) and strong evals practice (golden sets, LLM-as-judge).
• SQL proficiency for writing hirify.global queries and comfort with Docker and Kubernetes.

Nice to have

• Experience building production agents for observability, incident response, or SRE.
• Knowledge of prompt caching and context compaction techniques.
• Experience with columnar databases and event ingestion pipelines.
• Contributions to open-source AI or agent projects.
• Familiarity with Go, Rust, or other systems languages.

Culture & Benefits

• Globally distributed, remote-friendly work environment.
• Equity in the company through stock options for all new hires.
• Employer contributions toward healthcare.
• Flexible time off policy.
• $500 home office setup budget for remote employees.
• Opportunities for in-person connection via company-wide offsites.