Azure Databricks Data Engineer
Infosys
Infosys
Join our Data & Analytics practice to modernize data landscapes using cloud-native architectures, advanced analytics, and intelligent platforms. As a Principal Consultant specializing in Databricks Data Engineering, you will drive enterprise-scale data modernization programs. This includes leading Lakehouse implementations, cloud migrations, real-time analytics, and building AI-ready data ecosystems. You will leverage Databricks and cloud technologies to design, develop, and optimize cutting-edge data platforms.
Key responsibilities involve designing, developing, and maintaining robust data pipelines using Databricks and Apache Spark. You will build and optimize ETL/ELT workflows for large datasets, create data ingestion frameworks from diverse sources, and implement Lakehouse architectures with Delta Lake. Performance tuning of Spark jobs and Databricks workloads for scalability and cost efficiency is crucial. Collaborating with analytics, BI, and business teams to deliver reusable data assets and establishing data quality, monitoring, and governance controls are also part of this role. Troubleshooting production issues and contributing to architecture and code reviews will ensure best practices and business continuity.
This role demands extensive experience, with a minimum of 11 years of hands-on expertise in Databricks. Proficiency in PySpark or Spark SQL for distributed data processing and strong SQL skills with query optimization are essential. You should have proven experience in designing and developing ETL/ELT pipelines, along with practical knowledge of Delta Lake, DataFrames, and Spark architecture. A solid understanding of data warehousing, dimensional modeling, and data lake concepts is required. Experience with cloud platforms like Azure, AWS, or GCP, and familiarity with Git-based source code management and DevOps practices are necessary. Excellent analytical, debugging, and problem-solving skills are expected. A Bachelor of Engineering degree is required.
Infosys
Information Technology & Services