Track Manager - AWS IAC, Terraform,Python (IN)
HCLTech
HCLTech
We are seeking a skilled Production Evaluation and Observability Engineer with 6-9 years of experience to join our team. This role focuses on building a sophisticated production sampling sidecar, `syf-online-eval`, which closes the loop with `syf-eval`. The system will capture live sessions, perform asynchronous LLM-judge scoring, and provide drift and regression insights through a dedicated dashboard.
This position offers an exciting opportunity to work with cutting-edge observability tools and contribute to the continuous improvement of our evaluation systems. You will be instrumental in enhancing how we monitor and assess production performance.
Key responsibilities include developing the sampling sidecar to consume OpenTelemetry `gen_ai.*` traces from production agents, integrating asynchronous judge scoring, and building robust drift detection against agent golden-set baselines. You will also wire negative feedback signals into an evaluation queue and construct a comprehensive dashboard displaying usage, pass rates by behavior tags, drift alerts, and cost.
Furthermore, you will ensure adherence to data classification gating for sampled prompt and completion content, with the capture mechanism defaulting to OFF. This role demands a strong understanding of architectural patterns and data handling protocols.
To be successful in this role, you should possess at least 6 years of engineering experience, with a solid background in observability and monitoring pipelines. Proficiency with OpenTelemetry collectors and pipelines, along with experience using dashboard tools such as Splunk, New Relic, Grafana, or CloudWatch, is essential. A strong comfort level with asynchronous and sidecar architecture patterns is also required, as is a good understanding of data-classification-aware logging.
Familiarity with building LLM-judge-based drift/regression detection, Bedrock Guardrails, or similar content safety signals, and exposure to regulated industry monitoring requirements are considered advantageous.
HCLTech
IT Consulting