SDET Senior Test Lead (IN)
HCLTech
HCLTech
Seeking a seasoned QA Engineer with expertise in functional and automated testing, complemented by a practical grasp of AI and machine learning. This pivotal role ensures the quality, reliability, and trustworthiness of enterprise-grade AI solutions, including traditional ML models, Generative AI, RAG pipelines, agentic workflows, and their underlying data infrastructure. The ideal candidate possesses a deep understanding of AI systems' unique characteristics, where probabilistic outcomes, contextual correctness, and testing strategies must address model drift, hallucination, retrieval quality, and data integrity. Collaboration with AI Engineers, data engineers, and architects is essential for defining test strategies, constructing validation frameworks, and delivering insightful quality reports to diverse stakeholders.
This position is specifically tailored for individuals who understand the nuanced differences between testing AI and conventional software. You will be instrumental in shaping the quality assurance approach for cutting-edge AI applications.
Design and execute comprehensive test plans for AI/ML solutions, Generative AI, RAG pipelines, and agentic workflows. Validate end-to-end functional behavior and REST API interactions. Assess ML model outputs for accuracy, consistency, and adherence to expectations. Evaluate RAG pipeline quality, focusing on retrieval relevance, chunking, reranking, and LLM response accuracy. Test agentic AI workflows for correct tool invocation, reasoning, and error handling.
Perform data pipeline testing for ingestion, transformation, and quality. Execute regression testing for updates and changes. Test non-functional requirements like latency and scalability. Conduct exploratory testing to uncover emergent AI behaviors. Design and implement automated test frameworks using Python and frameworks like pytest. Build reusable automation scripts and integrate them into CI/CD pipelines. Automate Generative AI output evaluations and data quality checks. Integrate test automation with monitoring tools.
Evaluate LLM outputs for hallucinations, factual accuracy, and safety. Validate embedding quality and vector search in RAG. Test model fairness and robustness. Ensure responsible AI principles are functional. Support model drift monitoring. Produce clear test reports for technical and non-technical audiences. Maintain quality dashboards and metrics. Document defects thoroughly with reproducible steps and evidence.
Proven experience in functional, integration, regression, and exploratory testing of complex systems. Strong analytical skills for requirement analysis and test strategy design. Expertise in testing REST APIs with tools like Postman. Familiarity with testing data-intensive applications and data quality validation. Experience in Agile/Scrum environments.
Hands-on experience building and maintaining automated test suites in Python, preferably with pytest. Experience integrating test automation into CI/CD pipelines. Ability to write clean, maintainable automation code. Experience with API and data validation automation.
Working knowledge of AI/ML concepts, including model training, evaluation, and deployment implications for testing. Familiarity with Generative AI, LLMs, RAG, embeddings, vector search, and agentic workflows. Ability to evaluate LLM output quality and identify issues like hallucinations and factual errors. Understanding of AI non-determinism and probabilistic outputs. Awareness of responsible AI principles.
Strong written communication skills for clear test reports and defect documentation. Ability to summarize technical findings for non-technical stakeholders. Experience maintaining test metrics and quality dashboards using tools like Azure DevOps or Jira. Comfort presenting quality status and risks in team meetings. Familiarity with Azure AI services (Azure OpenAI, AI Search, Databricks, ADLS). Experience with Git and collaborative development. Familiarity with logging and observability tools.
HCLTech
IT Consulting