As a Principal Site Reliability Engineer, you will be instrumental in designing and architecting robust infrastructure and services, ensuring peak reliability and functionality. Your role involves forecasting demand, managing capacity needs, and collaborating closely with software development teams to build scalable and dependable infrastructures. You'll exercise critical judgment in data collection to optimize operations and reliability, leveraging advanced knowledge for incident response and maintenance. A key aspect will be providing comprehensive health and performance reports, identifying automation opportunities, and communicating service details proactively. You will also provide support for technology and document incidents, while continuously experimenting with new tools and staying ahead of site reliability trends.

Key responsibilities include designing and architecting infrastructure for optimal reliability, forecasting demand, and managing capacity. You will collaborate with development teams to build scalable infrastructures and identify prototyping opportunities. In incident management, you will perform data collection, triage, and technical analysis, monitor service performance, and conduct root cause analyses. Automation is crucial, involving identifying opportunities, developing tools and scripts, and testing them. You will also write release notes, communicate service details, and articulate the impact of changes. Providing comprehensive operational support, participating in on-call shifts, and resolving technical issues across services are essential. Finally, you will conduct advanced experiments with new tools, drive improvements for performance bottlenecks, and maintain in-depth knowledge of site reliability trends.

A minimum of 5 years of hands-on experience is required in software development, site reliability engineering, monitoring, and automation, with proven ownership of production platforms. Essential skills include strong Python programming, experience with backend services, REST APIs, and event-driven workflows. You must have proven experience building monitoring and observability solutions, and integrating AI/LLM capabilities into applications for incident analysis. The ability to develop reusable agents, implement alert correlation, and develop automated remediation workflows with approval controls is crucial. Experience integrating monitoring platforms with incident management systems and a strong understanding of distributed systems, Linux, networking, databases, and application performance are necessary. Proficiency in software engineering practices like automated testing, code reviews, CI/CD, version control, and secure development is expected. Excellent communication and collaboration skills are vital for working with various teams to define requirements and onboard services.