Join Optum, a global leader in healthcare technology, dedicated to improving lives through innovative solutions. As an AI Quality Engineer, you will play a pivotal role in ensuring the accuracy, reliability, security, and safety of our advanced agentic AI systems. This is an opportunity to contribute to a culture of inclusion and excellence, working alongside talented peers and driving meaningful improvements in health outcomes.

Our team is focused on advancing health optimization on a global scale, and your work will directly impact the resources and care millions of people receive. We offer comprehensive benefits and ample career development prospects, empowering you to grow and make a significant impact.

As an AI Quality Engineer, your responsibilities will include:

- Developing and executing a comprehensive, risk-based test strategy encompassing functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents. - Building and maintaining robust automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows. - Conducting manual exploratory and scenario-based testing to identify issues with new capabilities, ambiguous requests, edge cases, and real-world failure modes. - Creating and managing representative test datasets, defining expected outcomes, and developing evaluation rubrics using a blend of deterministic assertions, LLM-as-judge, and human review. - Performing rigorous quality checks for every release and for significant changes to prompts, models, tools, or data sources, integrating repeatable tests into CI/CD pipelines as release gates. - Monitoring production quality through key metrics like task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback. - Detecting drift and quality regressions, establishing alert thresholds, investigating failures, and expanding test coverage based on production findings. - Evaluating critical risks, including prompt injection, sensitive data exposure, bias, and unauthorized or unsafe agent actions. - Publishing clear quality dashboards and release-readiness reports, while partnering closely with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions. - Adhering strictly to all applicable Company policies, procedures, and business directives.

To excel in this role, you will need:

- A Bachelor's degree in Computer Science or a related field, or equivalent practical experience. - A minimum of 5 years of experience in software quality engineering or test automation, with at least 2 years focused on AI/ML, generative AI, or LLM-based systems. - Hands-on experience evaluating LLM applications using test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review. - Proven experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests. - Familiarity with integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications. - Proficiency in using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues. - Strong Python skills and demonstrable experience building or extending test automation frameworks like Pytest or Playwright. - A solid understanding of QA methodologies, defect lifecycle management, Agile delivery, and the software development lifecycle. - Knowledge of RAG, prompt engineering, tool use, and multi-step agent workflows. - Excellent analytical, problem-solving, communication, and collaboration skills are essential.