Join Optum, a global leader in healthcare technology, dedicated to improving health outcomes for millions. We are seeking a Data Engineering Lead passionate about building scalable data solutions.

Our inclusive culture fosters collaboration with talented peers and offers comprehensive benefits and career development. Be part of a team that's advancing health optimization on a global scale, impacting communities through innovation and connection.

This role offers a unique opportunity to drive impactful changes in healthcare delivery by leveraging cutting-edge data technologies and architecture.

Design, construct, and maintain robust batch and streaming data pipelines on Azure and Databricks, adhering to the Medallion Architecture. Process diverse data formats using Azure Data Factory and Spark. Develop sophisticated transformations in PySpark and Spark SQL, including CDC and SCD handling.

Implement data governance, lineage, and security with Unity Catalog. Build and refine data quality frameworks for accuracy. Optimize Spark jobs and Delta tables for peak performance and cost-efficiency.

Champion CI/CD and DataOps practices. Troubleshoot production issues with root cause analysis. Collaborate with cross-functional teams and mentor junior engineers. Leverage AI tools to enhance development and documentation.

Possess a B.E./B.Tech./MCA or equivalent degree in a relevant technical field. Bring at least 6 years of data engineering experience, with a minimum of 3 years focused on Azure Databricks.

Demonstrate hands-on expertise with Delta Lake, Lakehouse architecture, and Unity Catalog. Showcase experience with various file formats and Azure services like ADF and ADLS Gen2. Solid understanding of data modeling, ETL/ELT, Git, CI/CD, and Agile methodologies is essential.

Proficiency in Python, PySpark, and SQL, coupled with strong problem-solving and communication skills, is required. Experience with views, materialized views, and streaming tables is also expected.