NTT DATA is looking for accomplished Data Engineers to join a significant banking data modernization initiative. This program involves migrating approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. You will be instrumental in refining, expanding, and managing the reusable migration frameworks developed during the Proof of Concept. Key activities include building Spark/PySpark migration pipelines, integrating cyber/tokenization services, implementing robust migration controls, and ensuring automated reconciliation. You will also focus on optimizing Iceberg storage and troubleshooting PB-scale migration challenges. This role demands extensive hands-on engineering expertise, a strong grasp of distributed data processing, and a proven ability to construct reliable, restartable, and high-performance data pipelines.

Drive the migration framework by hardening and extending a config-driven Python/PySpark solution capable of migrating thousands of Hadoop/Hive tables without custom code per table. Develop reusable components for Hive metadata/schema discovery and migration configuration. Implement dynamic migration strategies (CTAS/bulk, partition-wise, chunk-wise) based on table characteristics and tokenization needs. Integrate crucial Data Movement Controls such as checkpointing, restart/recovery, idempotency, retries, control totals, audit logging, and exception handling.

Execute multi-TB/PB-scale Hadoop/Hive to Apache Iceberg historical migrations. Design effective migration patterns for tables of all sizes. Optimize parallel Spark execution across tables, partitions, and chunks. Implement resumability for large-table migrations to restart from safe checkpoints. Diagnose and optimize bottlenecks across Hadoop reads, Spark processing, tokenization, and Iceberg writes. Support migration wave execution and resolve production issues.

Integrate approved tokenization capabilities (e.g., TaaS/Protegrity) into migration pipelines, ensuring secure processing of sensitive fields and handling service disruptions. Measure and optimize tokenization's impact on migration throughput. Support technical validation of cyber/tokenization outcomes.

Configure and create Apache Iceberg tables according to established standards, implementing appropriate partitioning and file-sizing strategies. Optimize compaction and small-file handling. Work with Iceberg snapshots, schema evolution, and metadata. Troubleshoot Iceberg write, commit, and catalog issues. Support ongoing table maintenance and performance tuning.

Develop automated source-target validation covering row counts, schema conformity, partition completeness, control totals, aggregates/checksums, null/duplicate checks, and tokenization validation. Isolate reconciliation failures at the table/partition level. Implement selective retry/reprocessing for failed data. Produce auditable migration evidence.

Monitor and troubleshoot distributed Spark jobs, analyzing Spark UI, executor logs, and resource utilization. Resolve issues across Hadoop/HDFS, Spark, Iceberg catalog, object storage, and tokenization. Build operational telemetry for migration status, throughput, retries, and failures. Participate in code reviews, testing, CI/CD, and adhere to engineering standards. Mentor mid-level and junior Data Engineers.

Requires a minimum of 10 years of Data Engineering experience, with at least 4 years of hands-on production experience using Apache Spark/PySpark. Strong practical experience with Apache Iceberg is essential, including table creation, read/write operations, partitioning, snapshots, schema evolution, file sizing, compaction, and Spark integration.

Mandatory skills include proficiency in Python, PySpark, Apache Spark, Spark SQL, Hadoop, HDFS, Hive, HiveQL, Hive Metastore, SQL, Apache Iceberg, Parquet, Linux, and Git. You must have a solid understanding of distributed data processing, large-volume ETL/data migration, source-target reconciliation, and Spark performance tuning.

Deep understanding of parallelism, checkpoint/restart mechanisms, idempotency, retry/recovery strategies, large-table processing, small-file problems, and distributed system troubleshooting is crucial. Preferred skills include experience with Apache Polaris, Nessie, Glue, Kubernetes, OpenShift, Spark Operator, S3/object storage, TaaS/Protegrity, PostgreSQL, Trino, enterprise schedulers (Control-M/Airflow), CI/CD for Python/Spark, banking/financial services, PB-scale migrations, HiveQL to Spark SQL modernization, and lakehouse architectures (Bronze/Silver/Gold).