Azure Databricks Data Engineer
Infosys
Infosys
Join Infosys' Data & Analytics practice as a Lead Consultant and drive enterprise-scale data modernization programs. This role focuses on leveraging Databricks for implementing cloud-native architectures, advanced analytics, AI, and intelligent data platforms. You will be instrumental in modernizing data landscapes through Lakehouse implementations, cloud migrations, real-time analytics, and building AI-ready data ecosystems.
This opportunity is ideal for experienced professionals passionate about shaping the future of data within a leading technology organization. Contribute to cutting-edge projects and help clients unlock the full potential of their data.
As a Databricks Data Engineer, your core responsibilities include designing, developing, and optimizing modern data platforms. You will build and maintain scalable data pipelines using Databricks and Apache Spark, alongside creating robust ETL/ELT workflows for extensive data processing. Developing data ingestion frameworks from various sources and implementing Lakehouse architectures with Delta Lake are key. Optimizing Spark jobs for peak performance, scalability, and cost-efficiency is also critical.
Further duties involve collaborating with analytics and business teams to deliver reusable data assets, establishing data quality and governance controls, and troubleshooting production issues. You'll also participate in code and architecture reviews, promoting best practices, and creating essential technical documentation.
We are seeking candidates with 9-11 years of hands-on experience with Databricks. Essential skills include proficiency in PySpark or Spark SQL for distributed data processing, strong SQL expertise with query optimization, and proven experience in designing and developing ETL/ELT pipelines. Familiarity with Delta Lake, DataFrames, Spark architecture, and cloud platforms like Azure, AWS, or GCP is required.
Knowledge of data warehousing concepts, dimensional modeling, and data lake architectures is beneficial. Experience with Git-based source code management and DevOps practices is also important. Strong analytical, debugging, and problem-solving abilities are crucial for success in this role. A Bachelor of Engineering degree is a prerequisite.
Infosys
Information Technology & Services