Azure Databricks & PySpark Principal Consultant

Infosys

12–16 yrs Bengaluru Full Time Work from office
Infosys logo
Posted : today
Actively hiring

Job description

Seeking a Principal Consultant with extensive experience in Azure Databricks and PySpark to drive data engineering initiatives. This role involves building sophisticated, scalable data solutions on the Azure platform, processing vast datasets, and ensuring enterprise-grade delivery.

As a key member of the Data & Analytics Unit, you will contribute to Infosys Limited's commitment to innovation and excellence in data solutions.

Responsibilities

Lead the design, development, and optimization of large-scale data pipelines using Azure Databricks and PySpark.

Build and maintain robust ETL/ELT workflows with Azure Data Factory and develop scalable frameworks for data ingestion.

Implement complex data transformation, cleansing, and enrichment processes, working with massive datasets in Azure Data Lake.

Optimize Spark jobs for peak performance, scalability, and cost-effectiveness.

Collaborate closely with stakeholders, architects, and analytics teams to deliver impactful data solutions while ensuring data quality, security, and governance.

Qualifications

Requires 12-16 years of IT experience with a strong focus on Data Engineering, including deep hands-on expertise in Azure Databricks and PySpark development.

Demonstrated proficiency in Azure Data Factory, SQL, Data Warehousing concepts, and Azure Data Lake Storage (ADLS Gen2).

Solid understanding of Spark architecture and performance tuning is essential.

Experience designing and implementing enterprise-scale data pipelines, coupled with knowledge of CI/CD and Agile methodologies, is crucial for success in this role.

Essential Skills

Azure DatabricksPySparkAzure Data FactorySQLData WarehousingAzure Data Lake Storage Gen2Spark architecturePerformance tuningEnterprise-scale data pipelinesCI/CDAgile delivery

Good to Have

Azure Bastion

Highlights

  • Actively hiring

More Details

RoleAzure Databricks & PySpark Principal Consultant
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Work from office

About the Company

Infosys logo

Infosys

IT Consulting

Azure Databricks & PySpark Principal Consultant at Infosys | SkillMX | SkillMX