Senior Technical Specialist (IN)
HCLTech
HCLTech
This senior role focuses on the hands-on creation of robust data pipelines within Microsoft Fabric. You will architect and implement solutions using Data Factory pipelines, Copy activities, and Spark notebooks, adhering to a medallion architecture. Responsibilities include writing production-grade PySpark and SQL for data ingestion, transformation, cleansing, and star-schema modelling in Delta Lake/OneLake.
You'll also build and optimize semantic models with Direct Lake, DAX measures, and row-level security, ensuring peak performance. The role involves designing adaptable, metadata-driven ingestion frameworks and implementing sophisticated handling for incremental loading, schema drift, and data quality. Integrating Microsoft Purview for governance, including lineage and cataloguing, is a crucial aspect of this position.
Architect and build data pipelines in Microsoft Fabric, including Data Factory pipelines and Spark notebooks, following a bronze/silver/gold medallion architecture. Develop production-grade PySpark and SQL for data ingestion, transformation, cleansing, and star-schema modelling using Delta Lake/OneLake. Design and implement Direct Lake semantic models, optimizing performance with DAX measures and row-level security. Create metadata-driven ingestion frameworks for scalable data ingestion across multiple sources. Ensure robust handling of incremental loading, schema drift, and data quality issues.
Lead technical architecture, establishing design patterns, coding standards, and best practices. Drive code reviews, implement CI/CD for Fabric using Git, and enforce version control. Make and document key architectural decisions regarding build-vs-buy and trade-offs. Collaborate with governance, infrastructure, and security teams on workspace design, access, networking, and cost management.
Translate business requirements into a technical roadmap. Mentor and develop junior data engineers, providing design guidance and fostering Fabric expertise. Manage delivery within an Agile framework, balancing individual contributions with team support. Communicate project status, risks, and trade-offs effectively to all stakeholders.
Requires strong hands-on experience building data platforms on Microsoft Fabric, or equivalent experience with Azure Synapse/Databricks and Power BI. Expertise in PySpark and SQL, with proven production experience in Delta Lake and the medallion architecture is essential. Demonstrable experience in designing and building data pipelines, including both orchestration and transformation, with a clear understanding of when to utilize each. Proficiency in hands-on semantic modelling and DAX, ideally with Direct Lake, and experience in performance optimization.
Solid understanding of data warehousing principles and dimensional modelling techniques, including star schema and surrogate keys. Experience with incremental load patterns, schema drift handling, and data quality frameworks is expected. Familiarity with CI/CD and Git for data platforms, as well as workspace/environment management. A proven track record of technical leadership, including setting standards, conducting code reviews, and mentoring engineers, while remaining actively involved in hands-on development, is required.
Key skills include Python, PySpark, Medallion Architecture, Microsoft Fabric, Data Pipelines, and Data Engineering. Familiarity with the Azure ecosystem (ADLS, Key Vault, networking/Private Endpoints, Azure DevOps) is important. Exposure to Real-Time Intelligence/streaming (Eventstream, KQL) is beneficial. Domain experience in financial services/BFSI is desirable. Relevant certifications such as DP-600/DP-700 Fabric or Azure Data Engineer are a plus.
HCLTech
IT Consulting