Join our dynamic Engineering as a Service/Operate team in Bengaluru as a Senior Data Engineer I. This pivotal role supports Disney's Ad Technology big data platforms, focusing on enhancing reliability, scalability, and performance. You'll engage in hands-on development and operational tasks within Databricks, Spark, Flink, and Airflow (MWAA) environments. Your contributions will ensure our production data workflows are robust, cost-effective, and highly performant at scale.

This position offers a chance to significantly impact critical data infrastructure while collaborating with various platform, BI, and analytics teams. As a Senior Data Engineer I, you will also play a role in mentoring emerging talent and refining best practices under expert guidance.

Key responsibilities include designing, optimizing, and maintaining Spark and Flink pipelines within Databricks and Airflow, ensuring high reliability and availability of production data workflows. You will be instrumental in debugging and resolving issues across notebooks, jobs, and orchestration, as well as tuning Spark jobs and clusters for optimal performance and cost efficiency. Building and maintaining comprehensive monitoring dashboards, alerts, and metrics using tools like Datadog and CloudWatch is crucial. You will actively participate in incident response, conduct Root Cause Analysis (RCA), and collaborate with cross-functional teams, documenting best practices and mentoring junior engineers.

We are looking for candidates with at least 4 years of experience in data engineering or big data platform operations. Essential hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in a production setting is required. A strong command of SQL and expertise in ETL/ELT pipeline development are vital. Familiarity with AWS services such as S3, Glue, Athena, EMR, and EKS for pipeline management, along with working knowledge of observability tools like Datadog, CloudWatch, or Prometheus, is expected. Proficiency in Python, Scala, or Java for development and automation, plus familiarity with infrastructure-as-code tools like Terraform, are necessary. Excellent collaboration and problem-solving skills are a must.