Test Analyst - AI Assurance
Hexaware
Hexaware
Join our team as a Test Analyst specializing in AI Assurance. This role focuses on ensuring the quality, safety, reliability, and responsible behavior of advanced AI systems.
You will play a crucial part in developing and executing comprehensive test strategies for Generative AI, including RAG and agent-based architectures. This involves hands-on evaluation using industry-leading AI frameworks and platforms, alongside driving Responsible AI validation and insightful metric reporting.
Develop and implement AI Assurance test strategies to rigorously assess AI systems for quality, safety, reliability, and responsible AI behavior. Design AI evaluation strategies for LLMs, RAG, and agent-based systems, covering both functional and non-functional aspects. Contribute to golden dataset curation and synthetic dataset generation. Implement and utilize AI evaluation frameworks like Ragas and DeepEval, and leverage AI evaluation tools such as LangSmith, Arize, and LangFuse.
Instrument systems to establish AI observability and tracing using OpenTelemetry. Integrate AI evaluation processes into CI/CD pipelines. Conduct Responsible AI validation and report key RAI metrics. Analyze root causes for AI system issues and ensure adherence to AI regulatory considerations.
A strong understanding of GenAI system architecture, particularly RAG and agent-based systems, is essential. Proficiency in Python and AI evaluation SDKs is required. Experience with AI evaluation frameworks like Ragas and DeepEval, along with model and agent evaluation capabilities, is crucial.
Demonstrated ability in Responsible AI (RAI) validation, RAI metrics reporting, golden dataset curation, and synthetic dataset generation. Familiarity with AI observability, AI tracing, and OpenTelemetry (OTEL) is also key. Experience in AI quality engineering, AI governance, AI monitoring, AI testing, and assurance, including root cause analysis for AI systems, is expected. A working knowledge of AI regulatory considerations is beneficial.
Hexaware Technologies
IT Consulting