Join EY's Enterprise Risk - Regulatory Compliance team as a Senior GxP AI/Model Consultant. You will guide clients in understanding their business needs and implementing solutions aligned with EY methodologies. This role involves establishing and strengthening client relationships, ensuring high-quality deliverables, driving operational efficiency, and proactively managing risks.

We are looking for a detail-oriented AI Model Evaluation Consultant with substantial experience in AI/ML models, Generative AI (GenAI), and multi-agent workflows within GxP-regulated environments. Your primary focus will be to ensure the reliability, fairness, robustness, and regulatory compliance of AI solutions. You will collaborate closely with cross-functional teams to design, test, and maintain production-ready, inspection-ready AI systems.

This position will lead AI model consultation and development across the entire lifecycle, from integration and training to deployment and post-deployment monitoring. Adherence to GxP principles, risk-based validation, and data integrity expectations is paramount. You will embed Responsible AI, quality-by-design, and governance-by-design practices to enable the safe, trustworthy, and explainable use of AI and agent-based solutions in life sciences and pharmaceutical contexts.

Develop AI/ML and Agentic AI strategies, offering analysis, validation insights, and readiness assessments. Perform comprehensive AI/ML model integration, evaluation, and validation throughout development, training, testing, and deployment. Partner with data scientists and engineers to identify areas for model improvement and implement risk mitigation strategies. Design and execute robust testing strategies for AI agents and multi-agent workflows, focusing on orchestration logic, tool usage, and output validation. Document evaluation findings, limitations, risks, and acceptance criteria meticulously in an audit-ready format. Create or review essential validation documentation, including validation plans, user and functional requirements, risk assessments, traceability matrices, test protocols, test evidence, deviation records, and summary reports. Establish clear acceptance criteria for data quality and model performance, covering metrics like accuracy, sensitivity, specificity, precision, recall, F1 score, calibration, and error rates. Conduct or critically review functional, integration, regression, negative, boundary, security-role, audit-trail, data-reconciliation, and user-acceptance testing for AI-enabled workflows. Challenge model reliability through thorough testing for reproducibility, robustness, bias, subgroup performance, explainability, stress, and edge cases, clearly documenting limitations and residual risks. Define production monitoring frameworks for model performance, data and concept drift, anomalous outputs, override patterns, human review, incidents, and other signals requiring investigation or revalidation. Evaluate proposed model updates, retraining initiatives, prompt modifications, knowledge-base changes, infrastructure updates, and vendor releases through formal change control and impact assessment processes. Perform root cause analysis for issues such as hallucinations, inaccurate outputs, agent failures, latency problems, and unexpected model behavior. Assess model behavior across diverse datasets, edge cases, and failure scenarios to detect bias, drift, and instability.

A strong foundation in Machine Learning fundamentals and model evaluation techniques is essential. Demonstrate a solid grasp of classification, regression, clustering, anomaly detection, and key performance metrics (Precision, Recall, F1, AUC). Possess practical experience with LangChain, LangGraph, and Langfuse for agent workflows, observability, traceability, and evaluation logging. Gain hands-on experience evaluating LLM and GenAI-based applications and agents, focusing on accuracy, completeness, relevance, groundedness, faithfulness, consistency, and task completion. Analyze and investigate data drift, concept drift, model drift, and performance degradation in production environments. Understand retrieval pipelines for RAG-based solutions and be capable of evaluating retrieval quality, citation accuracy, and response grounding. Experience working with AI Observability Platforms such as Langfuse, LangSmith, Arize, WhyLabs, or MLflow for monitoring, tracing, evaluation logging, and dashboarding. Develop or review validation plans, user requirements, risk assessments, test protocols, and validation summary reports. Establish acceptance criteria for data quality and model performance, including key metrics. Execute or review various testing types for AI-enabled workflows, including functional, integration, and user-acceptance testing. Challenge model reliability through stress, edge-case, and bias testing, with clear documentation of limitations. Define production monitoring for model performance and drift. Evaluate proposed model updates and changes through formal impact assessments. Perform root cause analysis for issues like hallucinations and inaccurate outputs. Assess model behavior across datasets and failure scenarios to identify bias and drift. Possess a Bachelor’s or Master’s degree in Life Sciences, Engineering, or a related field. Accumulate 5+ years of experience in AI/ML, GenAI, software testing, validation, or quality engineering. Demonstrate experience in creating user stories, test scripts, validation plans, or AI solution prototypes. Have exposure to cloud ecosystems like Azure, AWS, or GCP. Exhibit excellent communication, documentation, and stakeholder management skills. Familiarity with version control and prompt engineering fundamentals is preferred. Understand data integrity principles and electronic records/e-signatures compliance. Previous client-facing experience is beneficial.