NTT DATA seeks a seasoned Senior Data Engineer/Technical Lead for a significant banking data modernization initiative. This role involves migrating approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The position demands a hands-on engineer capable of leading the engineering team while actively designing, developing, and optimizing solutions. You will spearhead the technical execution of the migration framework, Spark/PySpark development, Iceberg implementation, data modeling, Cyber/TaaS integration, migration controls, reconciliation, performance tuning, and the technical delivery of migration waves.

This is a hands-on leadership role, not solely a coordination position. You will contribute to code, conduct reviews, resolve complex production issues, and mentor junior Data Engineers.

Lead a Data Engineering team through assessment, development, migration, validation, and production execution. Translate architectural and migration standards into detailed engineering designs and reusable patterns. Own technical estimations, backlog management, work allocation, and technical delivery. Conduct design and code reviews, ensuring adherence to engineering standards. Mentor Data Engineers and provide direct technical guidance. Serve as the primary technical escalation point for complex migration and production issues. Collaborate with Data Architects, Technical PMs, security, infrastructure, and client technical teams. Lead root-cause analysis and ensure fixes are integrated into reusable framework patterns. Oversee the technical execution of migrating 2 PB of Hadoop/Hive data to Apache Iceberg. Enhance the reusable, configuration-driven migration framework. Analyze Hive/HDFS metadata, tables, and partitions to assess migration complexity. Implement migration strategies including bulk, partition-wise, and chunk-wise processing. Design execution approaches for various table sizes. Optimize parallel migration processes. Implement checkpointing, restart/recovery, idempotency, and safe retries. Lead technical execution and troubleshooting of migration waves.

Minimum 10 years of Data Engineering experience, including: - 5+ years of hands-on Apache Spark/PySpark experience. - Strong experience with Hadoop/Hive. - Proven experience leading Data Engineering teams. - Extensive experience in large-scale data migration and data modernization projects. - Solid background in data modeling and source-to-target mapping.

Multi-hundred-TB/PB-scale migration experience is highly preferred.

Mandatory Skills: - Python, PySpark, Apache Spark, Spark SQL, Hadoop, HDFS, Hive, HiveQL, Hive Metastore, SQL, Apache Iceberg. - Data modelling, Source-to-Target Mapping, Parquet, Linux, Git. - Large-volume ETL/Data Migration, Data Validation/Reconciliation. - Spark Performance Tuning, Distributed System Troubleshooting.

Data Modelling Experience: - Logical and physical data modelling. - Schema and data-type mapping. - Entity relationships, keys, and referential relationships. - Dimensional/relational modelling concepts. - Large-scale analytical data models.

Technical Leadership Experience: - Leading Data Engineering teams. - Technical estimation and planning. - Detailed solution design. - Code and design reviews. - Mentoring engineers. - Production troubleshooting and root-cause analysis. - Communicating technical issues with architects and client teams.