Embark on a career shaping the future of AI at EY, where global scale, supportive culture, and cutting-edge technology empower you to excel. We seek an experienced AI QA Senior with 5-9+ years in AI testing, quality engineering, and validation frameworks for enterprise AI solutions. This role is crucial for ensuring the quality, reliability, security, and performance of AI systems, including Agentic AI, RAG applications, and conversational AI.

Your expertise will be instrumental in collaborating with AI engineers, architects, and developers to establish robust quality assurance practices for AI and Generative AI implementations. A strong track record of professional implementation experience in relevant AI technologies is essential, with the ability to articulate your direct contributions to successful AI projects.

Showcase your capabilities by clearly explaining at least two relevant AI implementations, detailing the business use case, your specific role, the technical approach, and the delivered outcomes. This is an opportunity to build an exceptional career and contribute to a better working world.

Design and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems, ensuring the accuracy, consistency, and safety of Large Language Model outputs. Develop robust test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications.

Evaluate RAG systems, retrieval pipelines, and knowledge-based AI applications using defined frameworks. Conduct thorough functional, integration, regression, performance, reliability, and scalability testing of AI-powered applications. Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines.

Validate prompt engineering, agent workflows, tool usage, and reasoning in Agentic AI systems. Execute AI model testing across conversational, multimodal, and intelligent automation solutions. Perform critical security validations including AI penetration testing, adversarial testing, prompt injection, and jailbreak testing.

Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance. Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms. Utilize RAG evaluation frameworks like Ragas to measure retrieval quality and response effectiveness.

Develop testing dashboards, quality metrics, and defect reporting processes to drive continuous quality improvement. Collaborate with AI engineers and architects to identify quality risks and enhance operational reliability. Contribute to quality standards, governance controls, reusable frameworks, and best practices. Stay updated with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging QA technologies.

A minimum of 5–9+ years of overall professional experience is required, with at least 3 years of direct, hands-on experience in AI/ML, applied AI, or GenAI systems. You should have proven experience building production, pre-production, or enterprise pilot AI applications.

Demonstrate strong hands-on depth in at least three of the following areas, with meaningful ownership in at least two: LLM application development, retrieval pipelines, agent or tool orchestration, evaluation/guardrails/quality systems, model adaptation/fine-tuning, cloud deployment/optimization of AI applications, conversational AI, multimodal AI, or backend engineering for AI systems. A solid software engineering foundation for building reliable and maintainable AI services is crucial.

Educational qualifications include a Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field. Proficiency in testing AI-powered applications and enterprise AI platforms is expected, alongside experience in developing AI testing strategies, validation frameworks, and automated test suites.

Technical skills encompass a strong understanding of LLMs, Generative AI, Agentic AI, and Agentic RAG systems. Expertise in AI Model Testing, functional, regression, integration, and end-to-end quality validation is essential. Experience with AI penetration testing, adversarial testing, prompt injection, jailbreak testing, and vulnerability assessments is required.

Proficiency in prompt engineering validation, reasoning evaluation, and response quality assessment techniques is necessary. Implement evaluation frameworks for Agentic AI workflows and AI agents, with hands-on experience using Ragas or similar tools for assessing retrieval quality and response effectiveness. Validate Agentic RAG systems, retrieval pipelines, and knowledge-based AI applications.

Experience testing AI integrations across APIs, enterprise applications, databases, and cloud services is key. A strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking is expected. Familiarity with Microsoft Azure AI Platform and Azure AI services is beneficial.

Develop automated test frameworks and validation utilities using Python, with familiarity in FastAPI-based services and API testing frameworks. Design and execute automated API testing strategies. Knowledge of Responsible AI principles, AI safety controls, governance, and regulatory compliance is important.

Evaluate hallucinations, bias, toxicity, safety violations, and model robustness. Possess a strong understanding of software quality engineering, secure development practices, test automation frameworks, and CI/CD testing integration. Experience with performance testing, scalability testing, reliability testing, and production-readiness validation is required.

Familiarity with retrieval augmentation architectures, vector databases, embeddings, and enterprise search solutions is advantageous. Strong analytical, troubleshooting, and root-cause analysis skills are essential. Excellent problem-solving abilities and the capacity to connect AI capabilities to business value, coupled with strong communication and presentation skills, are vital.