Azure Databricks Data Engineer
Infosys
Infosys
Join our Data & Analytics practice to modernize data landscapes using cloud-native architectures, advanced analytics, AI, and intelligent data platforms. As a Senior Consultant, you will drive enterprise-scale data modernization programs, focusing on Lakehouse implementations, cloud migrations, real-time analytics, and building AI-ready data ecosystems.
This role is integral to leveraging Databricks and cloud technologies to design, develop, and optimize cutting-edge data solutions. You will contribute to building scalable, high-performance data platforms that empower enterprise-wide digital transformation.
Key responsibilities include designing, developing, and maintaining robust data pipelines using Databricks and Apache Spark. You will build and optimize ETL/ELT workflows for large datasets, develop data ingestion frameworks from diverse sources, and implement Lakehouse architecture with Delta Lake.
Further duties involve optimizing Spark jobs for performance and cost efficiency, collaborating with analytics and business teams to deliver data assets, and implementing data quality and governance controls. Troubleshooting production issues and participating in technical reviews to promote best practices are also essential.
This role requires creating technical documentation and reusable components, ensuring adherence to deployment standards.
We are seeking individuals with 7-9 years of hands-on experience with Databricks and proficiency in PySpark or Spark SQL for distributed data processing. Strong SQL skills, including query optimization and performance tuning, are essential.
Candidates should have experience designing and developing ETL/ELT pipelines, along with practical knowledge of Delta Lake, DataFrames, and Spark architecture. Familiarity with data warehousing concepts, dimensional modeling, and data lake architectures is required.
Experience with cloud platforms like Azure, AWS, or GCP, along with Git-based source code management and DevOps practices, is crucial. Excellent analytical, debugging, and problem-solving skills are necessary for success in this role.
Infosys
Information Technology & Services