Join Optum, a global leader dedicated to improving health outcomes through innovative technology. We are seeking a Data Engineering Consultant with expertise in ADB, ADF, Azure, Spark, and SQL to contribute to our mission of advancing health optimization worldwide. You will be part of a culture that values inclusion, collaboration, and continuous growth.

At Optum, we believe in creating a healthier future by connecting people with the care, resources, and data they need. Our team offers a supportive environment with comprehensive benefits and ample career development opportunities. Become an integral part of our journey to "Caring. Connecting. Growing together."

Design and construct robust, scalable data pipelines (ETL/ELT) for processing large datasets. Lead architectural decisions for data warehousing and big data solutions, focusing on scalability and efficiency. Optimize database performance, query execution, and overall data workflows to ensure high efficiency.

Implement comprehensive frameworks for data quality assurance, monitoring, and compliance. Collaborate closely with data scientists, analysts, and business stakeholders to facilitate advanced analytics and machine learning initiatives. Provide mentorship to junior engineers, fostering best practices in data engineering and promoting a culture of technical excellence.

Develop automated testing and deployment processes for data systems to streamline operations. Troubleshoot and resolve complex data-related issues, ensuring the reliability and integrity of all data systems. Adhere to all company policies, procedures, and employment contract terms.

A Bachelor's degree or equivalent practical experience is required. We are looking for at least 7 years of hands-on experience with Spark/Scala, Azure Databricks, Azure Data Factory, Airflow, and Kafka. Proficiency in coding, testing, implementing, debugging, and documenting complex Spark jobs is essential.

Demonstrated expertise in Azure Databricks, Python/Scala, and SQL is a must. Experience with ETL tools and big data frameworks like Spark and Hadoop is required, along with cloud platform experience, preferably Azure or GCP. Familiarity with containerization technologies such as Docker and Kubernetes is necessary.

Strong knowledge of data modeling, relational and NoSQL databases, and distributed systems, including data partitioning and parallel processing, is expected. Familiarity with data security and compliance standards like GDPR and HIPAA is also required. Preferred qualifications include experience with Snowflake, streaming technologies like Kafka, AI/ML deployment, CI/CD for data (Jenkins), and BI tools (Tableau, Power BI).