Seeking a Senior Azure Data Engineer / Sr. Technical Lead with profound expertise in Azure Databricks, PySpark, ADF, and SQL. This role is pivotal for designing, architecting, and delivering enterprise-scale data solutions. You will lead complex data initiatives, mentor engineering teams, and collaborate across departments to build a robust and scalable data ecosystem on Azure.

This position emphasizes strong technical leadership and hands-on engineering capabilities to drive modern data engineering practices and build a sophisticated data platform.

Spearhead the design, development, and deployment of large-scale data engineering solutions utilizing Azure Databricks, PySpark, SQL, and ADF. Architect comprehensive Modern Data Warehouse and Lakehouse solutions, prioritizing scalability, performance, security, and cost-efficiency.

Establish technical standards, best practices, reusable frameworks, and architectural guidelines for engineering teams. Provide technical leadership throughout the project lifecycle, from requirements analysis to deployment.

Develop, optimize, and maintain efficient ELT/ETL pipelines for processing large data volumes. Implement robust data ingestion frameworks for seamless integration with various sources. Ensure high availability and data reliability for orchestration workflows in Azure Data Factory.

Design and implement advanced transformation logic using PySpark on Databricks, leveraging Delta Lake features for data management. Optimize cluster performance, shuffle operations, caching, and job parallelization. Manage Databricks job pipelines, notebooks, clusters, scheduling, and CI/CD integration.

A minimum of 10–13 years of experience in data engineering and enterprise data platforms is required. This includes at least 3 years of hands-on experience with Azure Databricks (beyond proof-of-concepts) and a minimum of 5 years in building and orchestrating pipelines with Azure Data Factory (ADF).

Possess a minimum of 2+ years of strong PySpark experience, focusing on complex data transformation logic. A solid foundation of 6+ years in ETL and Data Warehouse, encompassing dimensional modeling, data partitioning, and performance optimization, is crucial.

Demonstrate strong SQL expertise, including complex query writing, optimization, stored procedures, and analytical functions. Proven experience working with Azure Data Lake Storage (ADLS), Delta Lake, and modern data processing patterns is essential. Experience in Agile/Scrum environments and knowledge of Azure Synapse, Event Hub, and Azure DevOps CI/CD are considered advantageous.