Test Analyst - AI Assurance
Hexaware
Hexaware
Seeking a skilled Test Analyst to join our AI Assurance team. This role focuses on evaluating Generative AI systems, including RAG and agent-based architectures. You'll utilize cutting-edge Python frameworks and platforms to ensure the quality, safety, and reliability of AI models. The position involves hands-on experience with AI evaluation tools, driving Responsible AI validation, and establishing robust AI observability through OpenTelemetry instrumentation. Your contributions will be vital for continuous improvement and effective monitoring of our AI systems.
Develop comprehensive AI Assurance test strategies to rigorously assess AI systems for quality, safety, reliability, and Responsible AI behavior. Design effective AI evaluation strategies for LLMs, RAG, and agent systems across both functional and non-functional aspects. Execute golden dataset curation and synthetic dataset generation for robust AI testing. Implement and leverage AI evaluation frameworks such as Ragas and DeepEval. Utilize AI evaluation tools including LangSmith, Arize, Langfuse, Galileo, and Evidently. Instrument systems to establish AI observability and tracing using OpenTelemetry libraries. Integrate AI evaluation processes into CI/CD pipelines for seamless deployment. Conduct Responsible AI validation to ensure ethical and unbiased AI outputs.
Demonstrated expertise in GenAI system architecture, particularly RAG and agentic systems. Strong proficiency in Python and experience with AI evaluation SDKs. Hands-on experience with AI evaluation frameworks like Ragas and DeepEval. Proven ability in model and agent evaluation, using tools such as LangSmith, Arize, and Langfuse. Experience with Responsible AI validation and reporting key RAI metrics. Skills in golden dataset curation and synthetic dataset generation. Proficiency in establishing AI observability and tracing with OpenTelemetry. Familiarity with integrating AI evaluations into CI/CD pipelines. Understanding of regulatory considerations for AI systems. Minimum of 4 years of experience in AI Assurance Testing.
Hexaware Technologies
IT Consulting