Join our team as a Sr. Technical Lead specializing in Data Engineering. This role focuses on architecting, developing, and optimizing advanced data platforms and analytical solutions within the Azure environment. You will be instrumental in building enterprise-grade data pipelines, managing data ingestion from various sources, implementing intricate transformations, and supporting scalable analytics initiatives.

This position offers a dynamic opportunity to shape modern data architectures and contribute to critical business intelligence efforts.

Key responsibilities include constructing and maintaining robust ingestion frameworks using ADF, Databricks, and Spark. You will implement data transformations from Bronze to Silver layers, aligned with architectural blueprints, and establish comprehensive data quality checks, schema validation, and contract rules.

Further duties involve building and optimizing high-throughput ELT/ETL pipelines for diverse data types, integrating data from on-premise and cloud systems, and developing complex, performant PySpark transformation scripts. You will also manage orchestration workflows in ADF, including pipelines, triggers, and integration runtimes.

Additional tasks include designing dimensional models (star/snowflake schemas), translating Silver datasets into analytics-ready formats with consistent KPI definitions, and ensuring reusability and scalability across data models and domains. Proficiency with Databricks Genie is beneficial.

Data governance and quality are paramount, involving the implementation of data quality checks, audit mechanisms, and validation frameworks within Unity Catalog. This includes enforcing Unity Catalog standards, managing metadata policies, and defining access controls.

You will also handle changes to column derivation logic, ensuring backward compatibility and conducting impact analyses. Partnering with business and data stewards to define business rules and validate metrics is also a crucial aspect of this role.

We are seeking a highly skilled Databricks Data Engineer with strong hands-on expertise in Databricks, PySpark, ADF, and SQL. Experience in building enterprise-grade data pipelines, ingesting data from diverse sources, and implementing complex transformations is essential.

This role requires the ability to build and optimize robust ELT/ETL pipelines, integrate data from multiple on-premise and cloud-based systems, and implement complex transformations using PySpark efficiently. Experience building orchestration workflows in ADF, including pipelines, triggers, and linked services, is necessary.

Familiarity with Databricks Genie is a plus. Experience building dimensional models, translating Silver datasets into analytics-ready formats, and ensuring consistency and reusability across data models is required.

You will develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations and implementing Delta Lake features like ACID transactions and schema evolution. Performance tuning and handling bottlenecks related to cluster configuration and joins are also part of the job. Collaboration with platform teams for Databricks cluster management and CI/CD integrations is expected.

Implementing data quality checks, audit mechanisms, and validation frameworks within Unity Catalog, along with enforcing its standards and access controls, is critical. You should be adept at handling changes to column derivation logic and conducting impact analysis.

Effective collaboration with customer IT and business teams, understanding data requirements, and delivering reliable solutions are key. You will also work with architects and product owners to align technical delivery with business objectives and provide guidance to junior engineers.