SME - Python (IN)

HCLTech

Fresher India Full Time Remote
HCLTech logo
Posted : today
Actively hiring

Job description

Join our dynamic team as a Senior Data Engineer, specializing in Databricks, PySpark, and Python. We are seeking an expert to architect and build robust, scalable data platforms and high-performance data pipelines. You will leverage the Databricks Lakehouse and implement advanced data architectures, including Medallion Architecture and Unity Catalog, while ensuring enterprise-level data governance.

Responsibilities

Your role will involve designing, developing, and maintaining scalable data pipelines and ETL/ELT solutions. You will build and optimize batch and streaming data processing using PySpark and Databricks, implementing the Medallion Architecture (Bronze, Silver, Gold layers). Configuration and management of Unity Catalog for centralized governance, access control, and data lineage are key. Developing data engineering solutions with Python and Spark frameworks, alongside optimizing Databricks workloads for performance and cost, will be central to your work. Collaboration with architects, analysts, data scientists, and business stakeholders is essential to deliver exceptional data solutions. You'll also ensure data quality, security, governance, and compliance across our enterprise data platforms, supporting Data Lake, Lakehouse, and Data Warehouse initiatives, and resolving complex data engineering issues.

Qualifications

We are looking for a candidate with strong hands-on experience in Databricks and its Lakehouse Platform. Expert-level proficiency in Python programming and deep expertise in PySpark and distributed data processing are crucial. A solid understanding of Big Data concepts and modern data architectures is required, along with practical experience implementing the Medallion Architecture. Experience with Unity Catalog for data governance, security, access management, and lineage tracking is essential. You should possess strong knowledge of Apache Spark performance tuning, experience in designing and building scalable ETL/ELT pipelines, and hands-on experience with Delta Lake and Lakehouse architecture. Familiarity with cloud platforms like Azure, AWS, or GCP is necessary. Excellent analytical, problem-solving, and communication skills are highly valued.

Essential Skills

DatabricksPySparkPythonBig DataMedallion ArchitectureUnity CatalogETLELTData PipelinesSparkDelta LakeCloud Platforms (AzureAWSGCP)

Highlights

  • Actively hiring

More Details

RoleSME - Python (IN)
DepartmentData Engineering
Employment TypeFull Time, Remote

About the Company

HCLTech logo

HCLTech

IT Consulting

SME - Python (IN) at HCLTech | SkillMX | SkillMX