NTT DATA is seeking a Senior Data Engineer / Technical Lead to spearhead a significant banking data modernization initiative. This role involves migrating approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The position demands a hands-on engineer with leadership capabilities, responsible for designing, developing, and optimizing the migration solution. Key accountabilities include managing the migration framework, Spark/PySpark development, Iceberg implementation, data modeling, cyber/tokenization integration, and ensuring the technical delivery of migration waves. This is not a purely coordination-focused role; active contribution to coding, design/code reviews, and complex issue resolution is expected, alongside mentoring junior engineers.

Lead a team of Data Engineers through all phases of assessment, development, migration, validation, and production execution. Translate architectural standards into detailed engineering designs and reusable patterns. Own the engineering backlog, work allocation, and technical delivery, ensuring adherence to engineering standards through design and code reviews. Provide hands-on technical guidance and mentorship to Data Engineers, acting as the primary escalation point for complex migration and production issues. Collaborate effectively with Data Architects, Technical PMs, security, and infrastructure teams.

Drive the technical execution of a 2 PB Hadoop/Hive to Apache Iceberg data migration. Enhance and extend the reusable migration framework. Analyze Hive/HDFS metadata, tables, and partitions to understand migration complexity. Implement various migration strategies, including bulk, partition-wise, and chunk-wise processing. Design approaches for tables of all sizes and optimize parallel migration. Implement checkpointing, restart/recovery, idempotency, and safe retry mechanisms. Lead troubleshooting efforts for migration waves.

Translate existing Hive data models into Apache Iceberg physical models, focusing on physical modeling for migration. Develop and review conceptual, logical, and physical data models as needed. Perform source-to-target data and schema mapping. Define and implement table structures aligned with Iceberg standards. Analyze entities, relationships, keys, data types, partitions, and business keys. Identify and resolve schema incompatibilities. Collaborate with the Data Architect on partitioning, file layout, and design decisions, ensuring downstream compatibility and preservation of business relationships.

Develop and review production-grade Python/PySpark components. Tune Spark jobs for PB-scale migration, diagnosing and resolving bottlenecks in Spark stages, executors, shuffle, skew, memory, and I/O. Optimize parallelism across tables and partitions and tune workloads to meet throughput objectives. Identify and coordinate remediation for bottlenecks across the entire migration pipeline (Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation).

Implement approved Apache Iceberg table standards. Develop and configure Iceberg tables and Spark-to-Iceberg processing. Implement partitioning, schema evolution, snapshots, file sizing, compaction, metadata maintenance, and small-file remediation. Troubleshoot Iceberg catalog, write, and commit issues, optimizing table performance and storage.

Integrate approved TaaS/Protegrity tokenization and retokenization patterns into Spark migration pipelines. Implement batching, retry, timeout, and recovery handling. Troubleshoot tokenization failures and performance bottlenecks, measuring their impact on migration throughput. Support Cyber/tokenization validation based on defined rules.

Lead the implementation of data movement controls, including checkpointing, idempotency, control totals, restart/recovery, retry mechanisms, audit logging, and exception handling. Build reusable source-to-target reconciliation capabilities covering row counts, schema validation, partition completeness, control totals, aggregates, checksums, null/duplicate validation, and tokenization validation. Isolate discrepancies at the table/partition level and enable selective reprocessing. Produce technical evidence for migration acceptance.

Requires a minimum of 10 years of Data Engineering experience, with at least 5 years of hands-on Apache Spark/PySpark expertise. Significant experience with Hadoop/Hive is essential. Proven experience in leading Data Engineering teams and managing large-scale data migration or modernization projects is required. Strong proficiency in data modeling and source-to-target mapping is critical.

Mandatory technical skills include Python, PySpark, Apache Spark, Spark SQL, Hadoop, HDFS, Hive, HiveQL, Hive Metastore, SQL, Apache Iceberg, data modeling, source-to-target mapping, Parquet, Linux, Git, large-volume ETL/data migration, data validation/reconciliation, Spark performance tuning, and distributed-system troubleshooting.

Experience with logical and physical data modeling, schema/data-type mapping, entity relationships, keys, referential relationships, dimensional/relational modeling concepts, and large-scale analytical data models is highly desirable.

Demonstrated experience in technical leadership, including team leadership, estimation, planning, solution design, code/design reviews, mentoring engineers, production troubleshooting, root-cause analysis, and communicating technical issues to stakeholders is crucial.