NTT DATA is seeking a skilled Databricks Engineer specializing in Systems Integration to join our dynamic team in Bangalore. This role is perfect for innovative individuals eager to contribute to an inclusive and forward-thinking organization. You will be instrumental in building and maintaining robust data pipelines, ensuring seamless integration of diverse data sources into production-ready datasets.

As a Databricks Engineer, you will design, construct, and maintain production data pipelines on Azure Databricks, utilizing PySpark and Delta Lake within a medallion architecture. Your responsibilities include ingesting data from various sources like Oracle, Netezza, and SQL Server using Azure Data Factory and Databricks Workflows, while managing incremental loads and schema drift.

You will develop reusable, configuration-driven frameworks for ingestion and transformation, embedding comprehensive testing and data quality checks into every pipeline. Ownership of CI/CD processes via Azure DevOps and Git, along with performance tuning and cost management, are key. You will also provide production support, conduct root-cause analysis for failures, and contribute to data modeling and solution architecture to support AI initiatives. Upholding engineering standards through code reviews and documenting all work thoroughly are essential aspects of this role.

We are looking for a professional with strong experience in data management, data classification, and data governance standards. Proficiency in leveraging AI-assisted development tools like GitHub Copilot for ETL development and code generation is expected. Hands-on experience in generating integrated test data across multiple sources is crucial.

Key technical skills include PySpark, Delta Lake, Azure Databricks, data pipelines, ETL processes, and data engineering. Familiarity with Azure Data Factory, Databricks Workflows, CI/CD practices, Azure DevOps, Git, data modeling, data quality, and cloud computing is necessary. Experience with system integration and a solid understanding of data governance policies are also vital for success in this position.