Join Optum, a global leader in healthcare technology, and contribute to improving health outcomes worldwide. We are seeking a talented Data Engineering Lead to design, develop, and deploy scalable data pipelines. You will be instrumental in connecting people with the care and resources they need to live healthier lives. This role offers a collaborative environment with talented peers, comprehensive benefits, and ample career development opportunities as we advance health optimization on a global scale.

This position is based in Bangalore, Karnataka, India, offering a hybrid work model. If you are passionate about making a difference and eager to grow with us, this is your opportunity to excel in a mission-driven organization. We encourage individuals from all backgrounds to apply and help us build a healthier future.

Spearhead the design, development, and deployment of robust data pipelines using Azure Databricks, Spark SQL, Python, and Airflow. Build and maintain efficient data ingestion, transformation, and curation processes from various enterprise data sources. Craft complex SQL queries and optimize workflows for peak performance and scalability. Develop and manage Airflow DAGs for seamless orchestration, scheduling, monitoring, and error handling. Collaborate closely with Product Owners and Business Analysts during requirement gathering and sprint planning.

Implement rigorous data quality validations and exception handling mechanisms. Support the evolution of Delta Lake and Lakehouse architectures within Databricks. Drive data platform modernization and cloud migration initiatives. Create reusable frameworks and automation solutions to boost engineering efficiency. Troubleshoot and resolve pipeline failures and performance bottlenecks. Adhere to coding standards, security requirements, and best practices. Contribute to CI/CD implementation for automated deployments.

Work cross-functionally to support analytics, reporting, AI, and business operations. Prepare comprehensive technical documentation and operational runbooks. Provide support for production deployments, incident resolution, and root cause analysis. Leverage AI-assisted development tools to enhance delivery efficiency. Offer technical recommendations and participate in architecture discussions. Demonstrate ownership and accountability for project deliverables. Stay abreast of emerging technologies and data engineering practices. Comply with all company policies, security standards, and directives.

A Bachelor's Degree in Engineering, Computer Science, Information Technology, MCA, or equivalent experience is required, coupled with a minimum of 4 years of hands-on experience in Azure Databricks and modern data engineering practices. Experience working within Agile/Scrum environments is essential.

Key technical proficiencies include Data Engineering expertise, Data Ingestion Frameworks, Data Transformation and Enrichment, Batch Processing Pipelines, Data Quality Implementation, Query Optimization, Performance Tuning, Metadata and Lineage Concepts, and a strong understanding of Cloud Technologies, particularly Microsoft Azure services such as Azure Data Factory, Azure Key Vault, and Azure DevOps or GitHub for CI/CD Pipelines.

Proven core skills encompass Azure Databricks, Apache Spark, Spark SQL, SQL Development, Python Programming, Apache Airflow for DAG Development and Workflow Orchestration, Delta Lake, Azure Data Lake Storage (ADLS), ETL/ELT Development, Data Warehousing Concepts, and Data Modeling Fundamentals. Solid analytical, problem-solving, debugging, and troubleshooting capabilities are expected, along with excellent communication and collaboration skills. A growth mindset, learning agility, and the ability to thrive in a dynamic environment are crucial.