Senior Azure Data Engineer ? Databricks, Kafka & Spark
Persistent
Persistent
Join Persistent as a Senior Azure Data Engineer and drive the design, development, and support of sophisticated data platforms for advanced healthcare analytics and digital transformation. This role demands expertise in Azure, Databricks, Kafka, Spark, and Airflow, crucial for building scalable, secure, and resilient data solutions. You'll collaborate with cross-functional teams to deliver modern data platforms and real-time processing capabilities, empowering enterprise-scale workloads.
This position offers an opportunity to work with industry leaders and contribute to innovative projects. We are committed to your growth, providing opportunities for professional development, continuous learning, and access to cutting-edge technologies. Persistent champions a values-driven, people-centric, and inclusive work environment.
In this role, you will design, develop, and maintain robust data engineering solutions on Azure and other cloud platforms. You will be instrumental in building and optimizing scalable data pipelines utilizing Databricks, Spark, Kafka, and Airflow for both batch and real-time processing. Developing and managing data integration frameworks, working with various database types, and implementing RESTful API integrations will be key. You'll also be involved in implementing data ingestion, transformation, orchestration, and quality processes.
Further responsibilities include supporting Data as a Service (DaaS) and Container as a Service (CaaS) initiatives, building CI/CD pipelines, monitoring systems, and engaging in troubleshooting and root cause analysis. You will also contribute to code reviews, mentor junior engineers, and uphold engineering best practices.
We are seeking a Senior Azure Data Engineer with 6-10 years of experience in data engineering and cloud application development. Essential skills include strong hands-on experience with Azure or AWS, a minimum of 5 years with Databricks and Kafka, and deep expertise in Apache Spark. Proficiency in building scalable batch and streaming data pipelines, along with hands-on experience with Apache Airflow for workflow orchestration, is required.
Experience with Snowflake data warehouse solutions, a solid understanding of Data Lake, Data Warehouse, and Lakehouse architectures, and proficiency in SQL with performance optimization are also crucial. Working knowledge of RESTful APIs, enterprise integration patterns, data modeling, governance, and quality frameworks is expected. Experience with CI/CD pipelines, DevOps practices, and familiarity with containerization technologies like Docker and Kubernetes are advantageous. Strong analytical, troubleshooting, and communication skills are essential for success in this role.
Persistent Systems
IT Consulting