Evaluation Framework Engineer
Persistent
Persistent
Join a leading digital engineering partner as an Evaluation Framework Engineer. This role is pivotal in shaping the future of AI evaluation for cutting-edge technologies like Large Language Models (LLMs), Agentic AI systems, and Generative AI applications. Leverage your Python expertise and quality-focused approach to build robust, scalable platforms.
This position offers a chance to work with industry-leading organizations and contribute to innovations across various sectors. You'll be part of a dynamic environment that values agility and client success, with numerous awards recognizing our leadership and commitment to excellence.
Design and implement advanced AI evaluation frameworks and essential tooling for assessing, benchmarking, and automating tests for LLMs, agents, and GenAI features. Develop production-ready Python applications and APIs to streamline evaluation pipelines and facilitate integrations.
Collaborate closely with the QA team to identify and address current evaluation challenges, establishing reproducible workflows. Implement comprehensive evaluation pipelines encompassing data preprocessing, metric calculation, test orchestration, and detailed reporting. Ensure superior code quality through adherence to coding standards, static analysis, unit/integration tests, and code reviews. Contribute to the design and implementation of APIs and services, deploying and managing evaluation components on Azure using infrastructure as code principles. Implement robust monitoring, logging, and alerting for evaluation pipelines, capturing audit trails and results for compliance and reproducibility. Partner with data scientists, ML engineers, and product stakeholders to refine requirements, validate evaluation strategies, and integrate feedback.
We are seeking a Python Engineer with 4-6 years of professional experience, ideally with exposure to QA practices. The ideal candidate possesses strong Python engineering skills, a keen QA mindset, and practical familiarity with Generative AI/LLM concepts and Azure cloud services. You will be instrumental in designing, building, and maintaining scalable evaluation systems while collaborating closely with QA teams and stakeholders.
Required qualifications include strong, demonstrable hands-on experience in Python development, a solid grasp of OOPs concepts, software design principles, and coding best practices. Experience with test-driven development, writing unit and integration tests, and collaborating with QA on automated testing is crucial. Familiarity with the full project lifecycle, including requirements, design, development, code review, deployment, maintenance, and deprecation, is expected. Experience building RESTful APIs using FastAPI, Flask, or Django, along with practical experience with Azure cloud services and deployment patterns (App Services, AKS, Azure Functions, Blob/Storage, DevOps pipelines) is essential. Exposure to CI/CD tooling and code quality tools like SonarQube is required. A working knowledge of AI/DS concepts, particularly GenAI, LLMs, RAG patterns, and agent architectures, is necessary. Strong problem-solving and debugging skills are vital, as is the ability to work effectively across distributed systems and communicate clearly with cross-functional teams.
Persistent Systems
Information Technology & Services