NTT DATA is seeking a skilled Databricks Engineer specializing in Systems Integration. This role is an opportunity to join an innovative and forward-thinking organization. We are looking for passionate individuals eager to grow with us and contribute to an inclusive and adaptable team.

This position is based in Bangalore, Karnataka, India, and offers a hybrid work model. Join us to be part of impactful projects and drive the future of data engineering.

Design, build, and maintain robust production data pipelines on Azure Databricks. Ingest data from diverse sources using ADF and Databricks Workflows, ensuring efficient handling of incremental loads and schema drift. Develop reusable, configuration-driven frameworks for data ingestion, transformation, and quality. Embed comprehensive testing and data quality checks into all pipelines, including unit tests, reconciliation, and automated DQ gates with clear failure handling and alerting.

Manage CI/CD processes using Azure DevOps and Git for repeatable, auditable deployments. Optimize pipeline performance and manage costs by diagnosing issues and configuring clusters effectively. Provide production support and reliability engineering, including root-cause analysis and implementing preventative measures. Contribute to data modeling, analysis, and solution architecture, designing data products aligned with enterprise standards and governance policies.

Conduct code reviews to uphold engineering and secure coding standards. Document all work thoroughly, including design documents, data dictionaries, and runbooks. Support UAT and production deployments with clear, proactive communication. Address and correct data quality issues. Coordinate daily with onsite and offshore teams for efficient development and continuous delivery. Leverage AI-assisted tools to enhance ETL development and automation.

Proven experience in designing and maintaining production data pipelines using PySpark, Delta Lake, and a medallion architecture on Azure Databricks. Expertise in data ingestion from various sources (Oracle, Netezza, MongoDB, SQL Server, Synapse, Azure Data Lake, APIs, files) via ADF and Databricks Workflows, managing CDC and late-arriving data.

Strong skills in building reusable, config-driven frameworks and embedding comprehensive testing and data quality measures. Experience with CI/CD, Azure DevOps, Git, Databricks Asset Bundles, and Infrastructure as Code. Proficiency in performance tuning, cost management, and reliability engineering for data pipelines.

Familiarity with data modeling, data analysis, and solution architecture. Knowledge of enterprise standards, governance policies, Unity Catalog, PII masking, and secure coding practices is essential. Experience in leveraging AI-assisted development tools like GitHub Copilot for ETL development and automation. Strong data management skills, including data classification, treatment, and governance. Excellent documentation and communication skills are required.