Join our Ad Technology team as a Senior Data Engineer to enhance the reliability, scalability, and performance of big data platforms and pipelines. This role involves hands-on development and operations for Databricks, Spark, Flink, and Airflow (MWAA) environments. You will ensure production data workflows are resilient, cost-efficient, and performant at scale.

Your contributions will focus on building, optimizing, and supporting data pipelines. You'll also play a key part in automation, monitoring, and incident response, collaborating with platform, BI, and analytics teams to ensure timely and accurate data delivery. As a Senior Data Engineer, you will mentor junior team members and help shape best practices under the guidance of leadership.

Key responsibilities include designing, maintaining, and optimizing Spark and Flink pipelines within Databricks and Airflow. You will ensure the reliability and availability of production data workflows, along with debugging and resolving issues in notebooks, jobs, and orchestration.

This role also involves tuning Spark jobs, clusters, and pipelines for optimal performance and cost efficiency, and analyzing logs to identify and resolve bottlenecks. You will build and maintain monitoring dashboards, alerts, and metrics using tools like Datadog or CloudWatch. Proactive monitoring of pipeline health and prompt escalation of issues are essential.

Additionally, you will participate in incident response, contributing to diagnosis and resolution of outages, and developing Root Cause Analyses (RCA) for long-term stability. Collaboration with data engineering, BI, and platform teams is crucial for successful delivery and operations. You will also document best practices, contribute to runbooks, and mentor junior engineers.

We are looking for candidates with at least 4 years of experience in data engineering or big data platform operations. Proven hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in production environments is required.

Strong SQL skills and experience in ETL/ELT pipeline development are essential. Familiarity with AWS services such as S3, Glue, Athena, EMR, and EKS for pipeline management is necessary. A working knowledge of observability tools like Datadog, CloudWatch, or Prometheus is expected.

Proficiency in Python, Scala, or Java for pipeline development and automation is key. Familiarity with infrastructure-as-code tools like Terraform is beneficial. Excellent collaboration and problem-solving abilities, especially within distributed teams, are highly valued. A Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent experience, is required.