Join NTT DATA as a Databricks Engineer specializing in Systems Integration. This role is instrumental in building and maintaining robust data pipelines within the Azure Databricks environment.

As a key member of our innovative team, you will contribute to accelerating client success and driving positive societal impact through responsible technology. We are seeking passionate individuals eager to grow with a forward-thinking organization.

This is an opportunity to work with cutting-edge technologies and influence the future of data solutions.

Design, develop, and manage end-to-end data pipelines on Azure Databricks using PySpark and Delta Lake with a medallion architecture.

Ingest data from a variety of sources including Oracle, Netezza, MongoDB, SQL Server, Synapse, Azure Data Lake, APIs, and file feeds, employing ADF and Databricks Workflows. This includes handling incremental loads, CDC, schema drift, and late-arriving data.

Develop reusable, configurable frameworks for data ingestion, transformation, and quality assurance to enhance team efficiency. Embed comprehensive testing, including unit and integration tests, source-to-target reconciliation, and automated data quality gates with clear failure handling and alerting.

Manage CI/CD processes and environment promotions using Azure DevOps/Git and Databricks Asset Bundles, ensuring repeatable and auditable deployments. Optimize performance and costs by diagnosing issues like skew and small files, fine-tuning Delta tables and cluster configurations.

Provide production support, conduct root-cause analysis for failures, and implement improvements to prevent recurrence. Contribute to data modeling, analysis, and solution architecture, translating business needs into well-structured datasets.

Design and build data products aligned with enterprise standards, governance, and compliance requirements to support AI model development. Conduct code reviews to maintain engineering standards, secure coding practices, and compliance, including Unity Catalog controls and PII masking.

Document all work comprehensively, including design documents, data dictionaries, lineage, and runbooks. Support pipeline demos, UAT, production deployments, and proactively communicate risks.

Collaborate daily with onsite and offshore teams to ensure efficient development and continuous delivery. Address and correct data quality issues, ensuring data accuracy and backfilling where necessary. Apply strong data management principles, including classification, treatment, and governance.

Leverage AI-assisted development tools like GitHub Copilot for ETL development and automation. Generate integrated test data across multiple sources for validation in lower environments.

Demonstrated experience in designing, building, and maintaining production data pipelines on Azure Databricks, incorporating PySpark, Delta Lake, and a bronze/silver/gold medallion architecture.

Proficiency in ingesting data from diverse sources (Oracle, Netezza, Mongo DB, SQL Server, Synapse, Azure Data Lake, APIs, file feeds) using ADF and Databricks Workflows, with expertise in handling incremental loads, CDC, schema drift, and late-arriving data.

Experience in developing reusable, config-driven frameworks and libraries for ingestion, transformation, and data quality, rather than one-off scripts.

Skills in embedding testing and data quality into pipelines, including unit/integration tests, source-to-target reconciliation, and automated DQ gates with failure handling and alerting.

Proficiency in CI/CD and environment promotion through Azure DevOps/Git, Databricks Asset Bundles, and infrastructure-as-code.

Ability to tune performance and manage costs by diagnosing skew, shuffle, and small-file issues, optimizing Delta tables and cluster configurations.

Experience in providing production support and reliability engineering, monitoring SLAs, performing root-cause analysis, and driving preventative design changes.

Competence in contributing to data modeling, data analysis, and solution architecture, including profiling sources, producing logical/physical and dimensional models.

Experience in designing and building data products aligned with enterprise standards, governance policies, and compliance requirements for AI model development.

Ability to conduct and act on code reviews, upholding engineering standards, secure coding practices, and compliance requirements like Unity Catalog access controls and PII masking.

Skills in documenting all work and supporting delivery, including design documents, data dictionaries, lineage, and runbooks, as well as pipeline demos and UAT support.

Strong experience in data management, including data classification, data treatment, and data governance standards and practices.

Proficiency in leveraging AI-assisted development tools such as GitHub Copilot for ETL development, data pipeline engineering, code generation, troubleshooting, and automation.

Hands-on experience in generating integrated test data across multiple data sources and tables to support lower-environment testing and validation.