Azure Data Engineer

Persistent

5–10 yrs Pune Full Time Hybrid (office + remote)
Persistent logo
Posted : today
Actively hiring

Job description

Join Persistent as a Senior Azure Data Engineer and contribute to building cutting-edge, enterprise-scale data platforms. Leverage your expertise in Azure, Databricks, Python, and SQL to develop robust solutions for analytics, reporting, and AI/ML initiatives. Collaborate with a dynamic team to drive innovation and deliver impactful data solutions for industry-leading organizations.

This role is ideal for experienced professionals passionate about cloud technologies and big data processing, seeking to make a significant impact in a growth-oriented environment.

Responsibilities

Design, develop, and optimize scalable data pipelines, ETL/ELT workflows, and enterprise data integration solutions using Azure services and modern cloud technologies. Build efficient data ingestion, transformation, and storage processes across diverse cloud and enterprise data platforms. Create reliable data engineering solutions leveraging Azure Databricks, Python, PySpark, and SQL. Partner with business stakeholders, data scientists, and engineering teams to define requirements and deliver high-quality data solutions. Implement comprehensive data quality, governance, security, and monitoring frameworks. Develop and maintain batch and real-time data processing systems, including Spark, Databricks, and Kafka. Construct and manage data warehouses, data marts, and analytical datasets to support enterprise reporting and decision-making. Aid AI/ML initiatives through effective feature engineering, data preparation, and scalable pipeline development. Implement CI/CD pipelines and automation frameworks, adhering to DevOps best practices. Monitor production systems, diagnose performance issues, and conduct thorough root cause analysis. Adhere to engineering best practices including code reviews, comprehensive documentation, testing, and meticulous release management. Contribute to Agile teams, ensuring the delivery of scalable and business-aligned data solutions.

Qualifications

A minimum of 5-10 years of professional experience in Data Engineering, ETL/ELT development, or Data Platform Engineering. Extensive hands-on experience with key Azure services such as Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, and Azure Data Lake Storage (ADLS). Advanced proficiency in SQL, including query optimization, performance tuning, and complex data transformations. Strong programming capabilities in Python, with practical experience developing enterprise-grade applications. Proven experience with PySpark, Apache Spark, and handling large-scale distributed data processing. Expertise in designing, developing, and maintaining scalable data pipelines and integration frameworks. Familiarity with both cloud-based and on-premises relational databases like SQL Server, Oracle, PostgreSQL, and Snowflake. Working knowledge of Kafka and streaming data architectures. A solid understanding of data modeling, data warehousing, master data management, and data governance principles. Experience implementing CI/CD pipelines, Git-based version control, DevOps practices, and Agile methodologies. Exceptional analytical, problem-solving, and troubleshooting skills. Demonstrated ability to collaborate effectively with cross-functional teams and business stakeholders. Excellent verbal and written communication skills.

Essential Skills

Azure Data FactoryAzure DatabricksAzure Synapse AnalyticsAzure Data Lake StorageSQLPythonPySparkApache SparkETLELTData WarehousingData ModelingData GovernanceCI/CDGitDevOpsAgile MethodologiesKafka

Good to Have

DatabricksAzurePySparkOraclePostgreSQLSnowflakeAI/ML

Highlights

  • Actively hiring

More Details

RoleAzure Data Engineer
IndustryHealthcare
DepartmentData Engineering, Software Development, Data Science, Data & Analytics
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Persistent Systems logo

Persistent Systems

Healthcare

Azure Data Engineer at Persistent | SkillMX | SkillMX