Track Lead - Extract Transform Load (ETL), Data Lakes (IN)
HCLTech
HCLTech
Join our team as a Senior Data Engineer, leading a crucial migration of an Oracle-based data warehouse to Databricks on the Allianz Data Platform (ADP). This role is ideal for experienced professionals skilled in modern data engineering practices and AI-enabling data platforms. You will play a key role in transforming our data infrastructure for advanced analytics and machine learning.
This is an opportunity to leverage your expertise in Databricks, SQL, and data warehousing to design and implement high-performance ETL/ELT pipelines. You will ensure data quality, optimize performance, and enable downstream AI/ML use cases, contributing significantly to our data strategy.
Key responsibilities include analyzing the existing Oracle data warehouse and PostgreSQL sources to design the target model in Databricks. You will build and optimize ETL/ELT pipelines using Spark SQL and PySpark to migrate core data warehouse tables and datamarts.
Translating Oracle SQL/PL-SQL logic into Databricks Spark SQL and PySpark, ensuring functional equivalence and performance optimization is crucial. You will also be responsible for ensuring the data quality, performance, and reliability of the new Databricks-based data warehouse. Collaboration with Enterprise Architects, the ADP platform team, and business stakeholders is essential for alignment with architecture, standards, and best practices. Documenting migration patterns, data models, and operational procedures will support long-term maintainability.
We are seeking candidates with 5-9 years of IT experience, possessing a graduate or postgraduate degree in Engineering or a postgraduate degree in non-engineering disciplines. Strong, hands-on experience with Databricks (Spark SQL, PySpark, Delta Lake, jobs, clusters) is a must.
Proven background in data warehousing, including dimensional modelling, ETL/ELT, and performance tuning, is essential. Solid experience with Oracle DWH (SQL, PL/SQL) and migrations from Oracle or similar legacy platforms is required. Experience integrating PostgreSQL as a data source into a central DWH/Lakehouse is also necessary.
An ability to reverse-engineer existing ETL logic (e.g., SAS jobs) and translate it into Databricks pipelines is important, even without direct SAS BI report experience. Strong communication and stakeholder management skills are vital for an international, multi-vendor setup. A good understanding of how well-modelled data supports advanced analytics and AI/ML capabilities is also expected. Strong analytical, logical, and communication skills (written and oral) are fundamental. A professional, innovative mindset with a drive for continuous improvement and a positive attitude are highly valued.
HCLTech
IT Consulting