Test Analyst  - AI Assurance

Hexaware

4+ yrs India Full Time Remote
Hexaware logo
Posted : today
Actively hiring

Job description

Join our team as a Test Analyst specializing in AI Assurance. This role focuses on ensuring the quality, safety, reliability, and responsible behavior of advanced AI systems.

You will play a crucial part in developing and executing comprehensive test strategies for Generative AI, including RAG and agent-based architectures. This involves hands-on evaluation using industry-leading AI frameworks and platforms, alongside driving Responsible AI validation and insightful metric reporting.

Responsibilities

Develop and implement AI Assurance test strategies to rigorously assess AI systems for quality, safety, reliability, and responsible AI behavior. Design AI evaluation strategies for LLMs, RAG, and agent-based systems, covering both functional and non-functional aspects. Contribute to golden dataset curation and synthetic dataset generation. Implement and utilize AI evaluation frameworks like Ragas and DeepEval, and leverage AI evaluation tools such as LangSmith, Arize, and LangFuse.

Instrument systems to establish AI observability and tracing using OpenTelemetry. Integrate AI evaluation processes into CI/CD pipelines. Conduct Responsible AI validation and report key RAI metrics. Analyze root causes for AI system issues and ensure adherence to AI regulatory considerations.

Qualifications

A strong understanding of GenAI system architecture, particularly RAG and agent-based systems, is essential. Proficiency in Python and AI evaluation SDKs is required. Experience with AI evaluation frameworks like Ragas and DeepEval, along with model and agent evaluation capabilities, is crucial.

Demonstrated ability in Responsible AI (RAI) validation, RAI metrics reporting, golden dataset curation, and synthetic dataset generation. Familiarity with AI observability, AI tracing, and OpenTelemetry (OTEL) is also key. Experience in AI quality engineering, AI governance, AI monitoring, AI testing, and assurance, including root cause analysis for AI systems, is expected. A working knowledge of AI regulatory considerations is beneficial.

Essential Skills

GenAI System ArchitectureRAGAgent-based SystemsPythonAI Evaluation FrameworksRagasDeepEvalModel Quality EvaluationAgent Quality EvaluationArizeLangFuseConfident AIResponsible AI (RAI) ValidationRAI MetricsGolden Dataset CurationSynthetic Dataset GenerationAI ObservabilityAI TracingOpenTelemetry (OTEL)LLM EvaluationAI Quality EngineeringAI GovernanceAI MonitoringAI TestingAI AssuranceRoot Cause Analysis (AI Systems)Regulatory Considerations for AIAI Evaluation SDKsCI/CD Pipeline Integration

Good to Have

LangSmithGalileoEvidently

Highlights

  • Actively hiring

More Details

RoleTest Analyst  - AI Assurance
DepartmentQuality Assurance
Employment TypeFull Time, Remote

About the Company

Hexaware Technologies logo

Hexaware Technologies

IT Consulting