Databricks Data Engineer
Infosys
Infosys
Join us as a Databricks Data Engineer to architect and build cutting-edge data platforms utilizing Databricks and cloud technologies. You'll focus on crafting robust, large-scale data pipelines and implementing Lakehouse architectures to power advanced analytics.
This role involves close collaboration with data architects, analysts, and business stakeholders, ensuring the delivery of scalable data solutions that drive enterprise digital transformation. You will be instrumental in modernizing data landscapes through cloud-native approaches, advanced analytics, and intelligent platforms.
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark. - Build and optimize ETL/ELT workflows for large-scale data processing. - Develop data ingestion frameworks for diverse data sources. - Implement and manage Lakehouse architectures with Databricks and Delta Lake. - Optimize Spark jobs and Databricks workloads for performance and cost. - Collaborate with teams to understand data needs and deliver reusable data assets. - Establish data quality, monitoring, and governance controls. - Troubleshoot production issues and ensure business continuity.
- Proven experience with Databricks and proficiency in PySpark or Spark SQL. - Strong SQL skills, including query optimization and performance tuning. - Hands-on experience designing and developing ETL/ELT pipelines. - Familiarity with Delta Lake, DataFrames, and Spark architecture. - Solid understanding of data warehousing, dimensional modeling, and data lake concepts. - Experience with cloud platforms like Azure, AWS, or GCP. - Knowledge of Git for source code management and DevOps practices. - Excellent analytical, debugging, and problem-solving abilities.
Infosys
Information Technology & Services