Seeking a Senior Azure Data Engineer / Sr. Technical Lead with deep expertise in Azure Databricks, PySpark, ADF, SQL, and modern data engineering. This role demands strong technical leadership and hands-on engineering to design, architect, and deliver enterprise-scale data solutions on Azure. The successful candidate will guide complex data initiatives, mentor teams, and collaborate with stakeholders to build a robust and scalable data ecosystem.

Key responsibilities include architecting and leading the development of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, and ADF. You will design end-to-end Modern Data Warehouse and Lakehouse solutions, focusing on scalability, performance, security, and cost. This includes defining technical standards, coding best practices, and architectural guidelines.

You will build, optimize, and maintain high-performance ELT/ETL pipelines for processing large datasets. This involves setting up complex data ingestion frameworks for seamless integration with various sources and ensuring resilient orchestration workflows in Azure Data Factory. Expertise in Azure Databricks and PySpark is crucial for designing advanced transformation logic, utilizing Delta Lake capabilities, and optimizing cluster performance. Managing Databricks job pipelines, notebooks, and clusters, along with CI/CD integration, will be part of your role.

Implementing robust data quality frameworks, enforcing security best practices, and maintaining detailed technical documentation are also key. This includes validation, reconciliation, error handling, metadata management, access control, encryption, and auditability. You will also prepare technical specification documents, interface designs, architecture diagrams, and operational runbooks.

Collaboration with BI, analytics, and business teams is essential to translate requirements into scalable solutions. You will lead code reviews, mentor junior engineers, and work closely with Scrum Masters and Product Owners in an Agile environment.

- Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, and ADF. - Architect end-to-end Modern Data Warehouse and Lakehouse solutions, ensuring scalability, performance, and security. - Define technical standards, best practices, and architectural guidelines. - Build and optimize scalable ELT/ETL pipelines for structured and unstructured data. - Set up complex data ingestion frameworks from on-premise, cloud, and third-party sources. - Ensure high availability and data reliability in Azure Data Factory workflows. - Design and implement advanced transformation logic using PySpark on Databricks. - Utilize Delta Lake features for efficient data management. - Optimize Databricks cluster performance and job parallelization. - Manage Databricks job pipelines, notebooks, clusters, and CI/CD integration. - Implement data quality frameworks, validation, and error handling. - Enforce security, access control, and data lineage best practices. - Prepare detailed technical documentation, including architecture diagrams and runbooks. - Collaborate with BI, analytics, and business teams to convert requirements into solutions. - Lead code reviews and provide technical guidance to engineers. - Work within an Agile delivery model.

- Extensive experience as a Senior Azure Data Engineer or Technical Lead. - Strong proficiency in Azure Databricks, PySpark, Azure Data Factory (ADF), and SQL. - Proven experience in designing and architecting Modern Data Warehouse and Lakehouse solutions. - Expertise in building, optimizing, and maintaining scalable ELT/ETL pipelines. - Experience with data ingestion frameworks and integrating diverse data sources. - Deep understanding of Delta Lake capabilities (ACID transactions, schema evolution, versioning). - Skilled in cluster-level tuning and job optimization on Databricks. - Experience with Databricks job pipelines, notebooks, and cluster management. - Knowledge of data governance, data quality frameworks, and metadata management. - Familiarity with security best practices for data platforms, including access control and encryption. - Ability to prepare comprehensive technical documentation. - Excellent collaboration and communication skills for stakeholder engagement. - Experience mentoring junior and mid-level engineers. - Experience working in an Agile development environment.