Senior Azure Data Engineer ? Databricks, Kafka & Spark

Persistent

6–10 yrs Pune Full Time Hybrid (office + remote)
Persistent logo
Posted : today
Actively hiring

Job description

Join Persistent as a Senior Azure Data Engineer and drive the design, development, and support of sophisticated data platforms for advanced healthcare analytics and digital transformation. This role demands expertise in Azure, Databricks, Kafka, Spark, and Airflow, crucial for building scalable, secure, and resilient data solutions. You'll collaborate with cross-functional teams to deliver modern data platforms and real-time processing capabilities, empowering enterprise-scale workloads.

This position offers an opportunity to work with industry leaders and contribute to innovative projects. We are committed to your growth, providing opportunities for professional development, continuous learning, and access to cutting-edge technologies. Persistent champions a values-driven, people-centric, and inclusive work environment.

Responsibilities

In this role, you will design, develop, and maintain robust data engineering solutions on Azure and other cloud platforms. You will be instrumental in building and optimizing scalable data pipelines utilizing Databricks, Spark, Kafka, and Airflow for both batch and real-time processing. Developing and managing data integration frameworks, working with various database types, and implementing RESTful API integrations will be key. You'll also be involved in implementing data ingestion, transformation, orchestration, and quality processes.

Further responsibilities include supporting Data as a Service (DaaS) and Container as a Service (CaaS) initiatives, building CI/CD pipelines, monitoring systems, and engaging in troubleshooting and root cause analysis. You will also contribute to code reviews, mentor junior engineers, and uphold engineering best practices.

Qualifications

We are seeking a Senior Azure Data Engineer with 6-10 years of experience in data engineering and cloud application development. Essential skills include strong hands-on experience with Azure or AWS, a minimum of 5 years with Databricks and Kafka, and deep expertise in Apache Spark. Proficiency in building scalable batch and streaming data pipelines, along with hands-on experience with Apache Airflow for workflow orchestration, is required.

Experience with Snowflake data warehouse solutions, a solid understanding of Data Lake, Data Warehouse, and Lakehouse architectures, and proficiency in SQL with performance optimization are also crucial. Working knowledge of RESTful APIs, enterprise integration patterns, data modeling, governance, and quality frameworks is expected. Experience with CI/CD pipelines, DevOps practices, and familiarity with containerization technologies like Docker and Kubernetes are advantageous. Strong analytical, troubleshooting, and communication skills are essential for success in this role.

Essential Skills

AzureDatabricksKafkaSparkAirflowSQLSnowflake

Good to Have

AWSRESTful APIsDockerKubernetes

Highlights

  • Actively hiring

More Details

RoleSenior Azure Data Engineer ? Databricks, Kafka & Spark
DepartmentData Engineering
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Persistent Systems logo

Persistent Systems

IT Consulting

Senior Azure Data Engineer ? Databricks, Kafka & Spark at Persistent | SkillMX | SkillMX