Azure Databricks Data Engineer
Infosys
Infosys
Join our Data & Analytics practice to modernize data landscapes through cloud-native architectures, advanced analytics, AI, and intelligent data platforms. As an Associate Consultant specializing in Databricks Data Engineering, you will contribute to enterprise-scale data modernization programs. This includes Lakehouse implementations, cloud migrations, real-time analytics, and building AI-ready data ecosystems.
This role offers an opportunity to work with cutting-edge technologies and drive significant digital transformation initiatives.
Key responsibilities include designing, developing, and maintaining robust data pipelines using Databricks and Apache Spark. You will build and optimize ETL/ELT workflows for extensive datasets, and develop data ingestion frameworks to integrate data from diverse sources.
Implementing and managing the Lakehouse architecture with Databricks and Delta Lake is crucial. Optimizing Spark jobs for performance and cost efficiency, and collaborating with analytics and business teams to deliver data assets are also core duties. You will implement data quality frameworks, monitor solutions, and troubleshoot production issues.
Furthermore, you will participate in architecture and code reviews, promote best practices, and create essential technical documentation and reusable components.
We are looking for professionals with 2-3 years of dedicated experience in Data Engineering and Big Data technologies, with strong hands-on expertise in Databricks. Proficiency in PySpark or Spark SQL for distributed data processing is essential.
Candidates should possess strong SQL skills, including query optimization and performance tuning. Experience in designing and developing ETL/ELT pipelines, along with hands-on work with Delta Lake, DataFrames, and Spark architecture, is required. Familiarity with cloud platforms like Azure, AWS, or GCP, and Git-based source code management and DevOps practices are also necessary.
A Bachelor of Engineering degree is required, alongside strong analytical, debugging, and problem-solving capabilities. Experience with data warehousing concepts, dimensional modeling, and data lake architectures is beneficial.
Infosys
IT Consulting