Lead the design and optimization of cutting-edge data processing frameworks on Azure Databricks. This Senior Consultant role involves advanced Spark tuning, implementing comprehensive data governance with Unity Catalog, and spearheading the adoption of modern Medallion architectures.

This is an exciting opportunity to drive data modernization and migration initiatives for clients. You will leverage your deep expertise to architect and deliver high-impact data solutions.

Design and implement scalable Medallion Architectures using Delta Live Tables and Databricks. Lead the optimization of complex Spark workloads to enhance execution speed and reduce cloud compute expenses. Implement enterprise-grade data governance and access controls via Unity Catalog. Architect automated CI/CD pipelines for Databricks using DABs or Terraform. Design sophisticated data integration patterns connecting Databricks, Azure Synapse, and various RDBMS. Provide technical mentorship to junior engineers through code reviews and best practice enforcement.

Possess 5-7 years of dedicated data engineering experience, with profound expertise in Azure Databricks and Apache Spark. Demonstrate advanced experience with Delta Lake and the Medallion Architecture (Bronze/Silver/Gold). Show a proven track record in Spark Performance Tuning, including handling data skew and shuffle tuning. Exhibit strong experience with Unity Catalog for robust data governance and security.

Key soft skills include strong leadership, advanced problem-solving for complex architectural challenges, and excellent communication for stakeholder interactions.