Data Engg with Databricks - Technical Lead-Data Engg
Birlasoft
Birlasoft
Seeking a skilled Azure Data Engineer with extensive experience in Azure Databricks, PySpark, Azure Data Factory, and SQL. This role focuses on architecting, developing, and enhancing modern data platforms and analytical solutions within the Azure ecosystem.
Key responsibilities include building robust enterprise-grade data pipelines, managing data ingestion from various sources, implementing complex transformations, and supporting scalable analytics initiatives to drive business insights.
- Design comprehensive data engineering solutions using Azure services like Data Factory, Databricks, PySpark, and SQL. - Architect and deploy scalable data warehouse and Lakehouse solutions on Azure Data Lake Storage and Databricks. - Develop data models, integration patterns, and reusable frameworks adhering to best practices. - Build and optimize high-performance ELT/ETL pipelines for structured, semi-structured, and unstructured data. - Integrate data from on-premise, cloud systems, APIs, and third-party sources. - Implement sophisticated transformations using PySpark with a focus on efficiency and modularity. - Create and manage orchestration workflows in Azure Data Factory. - Develop and optimize PySpark scripts on Databricks, leveraging Delta Lake features and performance tuning techniques. - Implement data quality checks, audit mechanisms, and validation frameworks. - Ensure adherence to data security, access control, and lifecycle management standards. - Collaborate with BI, analytics, and business teams to deliver data solutions that meet business needs. - Work with architects and product owners to align technical delivery with strategic objectives.
This role requires a strong background in Azure Data Engineering, specifically with Databricks, PySpark, Azure Data Factory, and SQL. You should have proven experience in designing and building enterprise-level data platforms and analytical solutions on Azure.
Your expertise will be crucial in developing scalable data pipelines, managing diverse data sources, and implementing complex data transformations. A solid understanding of data warehousing, Lakehouse architecture, and data governance principles is essential. Experience with performance tuning and collaborating with cross-functional teams is also expected.
Birlasoft
IT Consulting