Seeking a highly skilled Senior Technical Lead in Data Engineering with extensive experience in Databricks, PySpark, Azure Data Factory (ADF), and SQL.

This role focuses on architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. You will be instrumental in building enterprise-grade data pipelines for diverse data ingestion, complex transformations, and scalable analytics initiatives.

Key responsibilities include building robust ingestion frameworks using ADF, Databricks, and Spark. You will implement data transformations from Bronze to Silver layers, architect data quality checks, and develop high-throughput ELT/ETL pipelines. Integration of data from various on-premise and cloud systems, along with implementing complex PySpark transformations for performance and modularity, will be crucial.

Develop scalable PySpark transformation scripts on Databricks, leveraging advanced optimizations like caching, partitioning, and Delta Lake features such as ACID transactions and schema evolution. You will also be involved in performance tuning, cluster configuration, and collaborating with platform teams for CI/CD integrations. Ensuring data accuracy and consistency within Unity Catalog, enforcing governance standards, and managing data changes with backward compatibility analysis are vital aspects of this role.

This position requires strong hands-on expertise in Databricks, PySpark, ADF, and SQL. Experience in building enterprise-grade data pipelines, managing data ingestion from diverse sources, and implementing complex transformations is essential.

Familiarity with Databricks Genie is preferred. You should be adept at building dimensional models, translating datasets into analytics-ready formats, and ensuring consistency and reusability across data models. Collaboration with business and IT teams to understand requirements and deliver reliable solutions is expected, along with providing guidance to junior engineers.