NTT DATA is seeking seasoned Data Engineers to join a significant banking data modernization initiative. This role involves migrating approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The engineer will be instrumental in enhancing and operating reusable migration frameworks, developing Spark/PySpark migration pipelines, integrating cyber/tokenization services, and implementing robust migration controls and automated reconciliation. Optimization of Iceberg storage and troubleshooting PB-scale migration workloads are key aspects of this position.

This is a hands-on engineering role demanding a deep understanding of distributed data processing and expertise in building reliable, restartable, and high-performance data pipelines.

Key responsibilities include hardening and extending a config-driven Python/PySpark migration framework for large-scale table migrations, building reusable Hive metadata discovery components, and implementing diverse migration strategies. You will also integrate Data Movement Controls like checkpointing, restart/recovery, idempotency, retries, control totals, audit logging, and exception handling.

Execute Hadoop/Hive to Apache Iceberg historical migrations at PB scale, designing patterns for various table sizes and tuning parallel Spark execution. Implement resumability for failed migrations and diagnose/optimize bottlenecks across Hadoop reads, Spark processing, tokenization, and Iceberg writes. Support migration execution and production issue resolution.

Integrate and optimize tokenization capabilities for secure processing of sensitive fields, managing service failures and retries. Develop automated validation covering row counts, schema conformity, partition completeness, control totals, aggregates, null/duplicate checks, and tokenization outcomes. Isolate reconciliation failures, implement selective reprocessing, and produce auditable migration evidence.

We require a minimum of 10 years of Data Engineering experience, with at least 4 years of hands-on production experience using Apache Spark/PySpark. Essential skills include strong proficiency in Python, PySpark, Apache Spark, Spark SQL, Hadoop, HDFS, Hive, HiveQL, SQL, Apache Iceberg, Parquet, Linux, and Git.

Candidates must possess solid experience in distributed data processing, large-volume ETL/data migration, source-target reconciliation, and Spark performance tuning. Practical experience with Apache Iceberg functionalities such as table creation, partitioning, snapshots, schema evolution, file sizing, compaction, and Spark integration is crucial.

A strong understanding of parallelism, checkpoint/restart mechanisms, idempotency, retry/recovery processes, large-table processing, small-file problems, and distributed system troubleshooting is mandatory. Preferred skills include experience with various Iceberg catalogs, containerization technologies, object storage, tokenization solutions, schedulers, CI/CD practices, and experience within the banking or financial services sector.