We are seeking a skilled Databricks Data Engineer with significant expertise in Databricks, PySpark, Azure Data Factory (ADF), and SQL. This role focuses on designing, building, and optimizing modern data platforms and analytical solutions within the Azure ecosystem.

Your responsibilities will include developing enterprise-grade data pipelines that handle ingestion from various sources, complex data transformations, and support scalable analytics initiatives. A strong understanding of data architecture and best practices is essential for success in this position.

Key duties involve executing data engineering tasks such as building robust ingestion frameworks using ADF, Databricks, and Spark. You will implement Bronze to Silver transformations, establish data quality checks, schema validation, and contract rules. Constructing high-throughput ELT/ETL pipelines to manage structured, semi-structured, and unstructured data is crucial.

Integrating data from diverse on-premise and cloud systems, APIs, and third-party sources will be a core function. You will develop complex transformations with PySpark, focusing on performance and modularity. Building orchestration workflows in ADF, including pipelines, triggers, and parameterized datasets, is also a key responsibility.

Additionally, you will design and implement curated data models in the Gold layer, including dimensional models, fact tables, and handling slowly changing dimensions. Ensuring consistency across domains, reusability, and scalability of these models is paramount. You will leverage Databricks and PySpark for scalable transformations, applying optimizations and Delta Lake features like ACID transactions, schema enforcement, and time travel. Performance tuning and collaboration with platform teams for cluster management and CI/CD are also expected.

This role demands expertise in Data Governance and Quality, including implementing checks, audits, and validation frameworks within Unity Catalog. You must enforce Unity Catalog standards, naming conventions, metadata policies, and access controls (RBAC/ABAC).

Handling changes to column derivation logic, performing backward compatibility analysis, and partnering with business/data stewards to define business rules and validate metrics are essential. Strong collaboration skills are needed to work closely with customer IT and business teams, understanding data requirements and delivering reliable solutions. You will partner with architects, product owners, and engineering teams to align technical delivery with business goals and provide mentoring to junior engineers.