Technical Lead (IN)
HCLTech
HCLTech
Seeking a skilled Technical Lead to drive data engineering initiatives. This role requires extensive experience with Microsoft Fabric, including OneLake and the Medallion Architecture (Bronze, Silver, Gold layers). A strong foundation in PySpark, SQL, and data transformation pipelines is essential for success.
This position is ideal for candidates focused on building robust data solutions. Individuals concentrating solely on Semantic Models or Reporting Layers may not be the best fit for this specific opportunity.
Key responsibilities involve constructing the Bronze, Silver, and Gold data layers, developing reusable frameworks, and implementing comprehensive data quality and reconciliation controls.
You will build a metadata-driven ingestion and transformation framework, develop ingestion processes for Bronze, cleansing for Silver, and marts for Gold layers. Implementing data quality rules, quarantine handling, and reconciliation controls across all layers is crucial.
Additional duties include building orchestration, scheduling, and dependency management within Fabric pipelines, optimizing performance, partitioning, and capacity usage. Support for SIT, parallel runs, and hypercare, along with defect resolution, is expected. All code should be clean, version-controlled, and well-documented.
Essential skills include data engineering expertise on Spark with PySpark and Spark SQL, alongside hands-on experience with Microsoft Fabric (Lakehouse, Notebooks, Pipelines) or equivalent platforms like Synapse or Databricks.
Proficiency in SQL and a solid understanding of dimensional modeling are required. Familiarity with Git and CI/CD practices is also necessary for effective workflow management.
Candidates should possess experience in testing and debugging data pipelines to ensure reliability and efficiency.
HCLTech
IT Consulting