Azure Databricks & PySpark Principal Consultant
Infosys
Infosys
Seeking a Principal Consultant with extensive experience in Azure Databricks and PySpark to drive data engineering initiatives. This role involves building sophisticated, scalable data solutions on the Azure platform, processing vast datasets, and ensuring enterprise-grade delivery.
As a key member of the Data & Analytics Unit, you will contribute to Infosys Limited's commitment to innovation and excellence in data solutions.
Lead the design, development, and optimization of large-scale data pipelines using Azure Databricks and PySpark.
Build and maintain robust ETL/ELT workflows with Azure Data Factory and develop scalable frameworks for data ingestion.
Implement complex data transformation, cleansing, and enrichment processes, working with massive datasets in Azure Data Lake.
Optimize Spark jobs for peak performance, scalability, and cost-effectiveness.
Collaborate closely with stakeholders, architects, and analytics teams to deliver impactful data solutions while ensuring data quality, security, and governance.
Requires 12-16 years of IT experience with a strong focus on Data Engineering, including deep hands-on expertise in Azure Databricks and PySpark development.
Demonstrated proficiency in Azure Data Factory, SQL, Data Warehousing concepts, and Azure Data Lake Storage (ADLS Gen2).
Solid understanding of Spark architecture and performance tuning is essential.
Experience designing and implementing enterprise-scale data pipelines, coupled with knowledge of CI/CD and Agile methodologies, is crucial for success in this role.
Infosys
IT Consulting