Join Optum, a global leader dedicated to improving health outcomes through innovative technology. As a Data Engineering Consultant, you will play a pivotal role in shaping the future of healthcare data by designing and implementing robust data solutions. This role offers a dynamic environment within an inclusive culture, surrounded by talented peers and ample opportunities for career growth.

Contribute to advancing health optimization on a global scale and become part of a team that is passionate about Caring, Connecting, and Growing together. This is an opportunity to make a significant impact on the communities we serve by leveraging cutting-edge data engineering practices.

Design, develop, and deploy scalable data pipelines utilizing Azure Databricks, Spark SQL, Python, and Airflow. Build and maintain data ingestion, transformation, and curation processes from various enterprise sources. Develop complex SQL queries and optimize data processing for performance. Create and manage Airflow DAGs for robust workflow orchestration, scheduling, monitoring, and error handling.

Collaborate with stakeholders during requirement gathering and sprint planning. Implement data quality validations and exception handling. Support Delta Lake and Lakehouse architectures. Participate in data platform modernization and cloud migration projects. Develop reusable frameworks and automation solutions for engineering efficiency. Troubleshoot pipeline failures and performance bottlenecks. Follow coding standards and security best practices.

Contribute to CI/CD processes. Support analytics, reporting, AI, and operational needs. Prepare technical documentation and operational runbooks. Assist with production deployments and incident resolution. Utilize AI-assisted development tools for enhanced efficiency. Provide technical recommendations and participate in design reviews. Demonstrate ownership and accountability for project deliverables. Stay updated on emerging data engineering technologies and cloud services.

A Bachelor's Degree in Engineering, Computer Science, Information Technology, MCA, or equivalent experience is required. Candidates should possess over 4 years of hands-on experience in Azure Databricks and modern data engineering practices, preferably within Agile/Scrum environments.

Key technical expertise includes Data Engineering, data ingestion frameworks, data transformation, batch processing, data quality implementation, query optimization, and performance tuning. Proficiency in cloud technologies, particularly Microsoft Azure, Azure Data Factory (ADF), Azure Key Vault, and Azure DevOps or GitHub is essential. Strong skills in Azure Databricks, Apache Spark, Spark SQL, SQL Development, Python Programming, and Apache Airflow for DAG development and workflow orchestration are also crucial.

Experience with Delta Lake, Azure Data Lake Storage (ADLS), ETL/ELT development, data warehousing concepts, and data modeling fundamentals is necessary. Preferred qualifications include Azure Databricks certification, experience with healthcare data domains, Snowflake, modern cloud data platforms, and DevOps automation. A solid understanding of analytical problem-solving, debugging, communication, and teamwork is expected.