Track Lead - Extract Transform Load (ETL), Data Lakes (IN)
HCLTech
HCLTech
Lead the charge in modernizing our data infrastructure by migrating an existing Oracle data warehouse to Databricks on the Allianz Data Platform. This senior role requires deep expertise in Databricks, SQL, and data warehousing, with a focus on implementing cutting-edge data engineering practices.
We're looking for someone passionate about building AI-enabling data platforms and ensuring the highest standards of data quality and performance. Join us to drive innovation and shape the future of our data landscape.
Analyze the current Oracle data warehouse and PostgreSQL sources, designing the optimal target model within Databricks.
Develop and refine ETL/ELT pipelines using Spark SQL and PySpark to seamlessly migrate core data warehouse tables and datamarts.
Translate complex Oracle SQL/PL-SQL logic into efficient Databricks Spark SQL and PySpark code, ensuring functional parity and peak performance.
Maintain the integrity, performance, and reliability of the new Databricks data warehouse, empowering advanced analytics and AI/ML initiatives.
Collaborate effectively with Enterprise Architecture, the ADP platform team, and business stakeholders to uphold architectural standards and best practices.
Create comprehensive documentation for migration strategies, data models, and operational procedures to ensure long-term manageability and knowledge sharing.
Possess 5-9 years of experience in data engineering, with a strong foundation in data warehousing, ETL/ELT, and performance tuning.
Demonstrate advanced proficiency with Databricks, including Spark SQL, PySpark, Delta Lake, job orchestration, and cluster management.
Showcase solid experience in data warehousing principles, dimensional modeling, and working with Oracle DWH environments, including SQL and PL/SQL.
Experience integrating PostgreSQL as a data source into a central DWH/Lakehouse is essential.
Ability to reverse-engineer and translate existing ETL logic (e.g., SAS jobs) into Databricks pipelines is crucial.
Exceptional analytical, logical, and communication skills, coupled with strong stakeholder management abilities in a global, multi-vendor setting.
An understanding of how well-structured data fuels advanced analytics and AI/ML capabilities is highly valued.
HCLTech
IT Consulting