Senior Technical Lead (IN)

HCLTech

10+ yrs India Full Time Remote
HCLTech logo
Posted : today
Actively hiring

Job description

Join our team as a Core Data Engineer specializing in AWS. You will be instrumental in designing, building, and maintaining secure, reliable, and cost-effective data pipelines. This role involves handling batch, streaming, and Change Data Capture (CDC) ingestion, culminating in curated datasets prepared for advanced analytics. A strong emphasis will be placed on integrating testing, governance, observability, and CI/CD practices throughout the data lifecycle.

Responsibilities

Key responsibilities include implementing robust ingestion pipelines using AWS Lambda and EMR for batch and streaming data, managing schema evolution, and orchestrating workflows with Step Functions or MWAA (Airflow). You will model datasets on S3 using formats like Parquet and Iceberg, manage data quality with automated tests and CI gates, and implement comprehensive security and governance measures including IAM, KMS, and Lake Formation policies. Driving observability through CloudWatch and optimizing costs via FinOps principles are also crucial. This role requires provisioning infrastructure with IaC tools like Terraform and building CI/CD pipelines with GitHub Actions, alongside maintaining essential documentation and runbooks.

Qualifications

We are seeking candidates with over 10 years of experience in building data platforms on AWS, demonstrating strong proficiency in Python (including PySpark) and SQL. Hands-on experience with AWS services such as DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA, S3/Parquet/Iceberg, Glue Data Catalog, Athena, and Redshift/Serverless is essential. Expertise in Infrastructure as Code (Terraform) and CI/CD (GitHub Actions) is required, along with experience in data quality frameworks like Great Expectations/Deequ, schema evolution, and security/compliance (IAM/KMS, private networking, Lake Formation, audit/lineage). A proven ability to tune performance and costs across storage and compute is also necessary.

Essential Skills

PythonPySparkSQLAWSDMSKinesisMSKGlueEMRLambdaStep FunctionsMWAAS3ParquetIcebergGlue Data CatalogAthenaRedshiftRedshift ServerlessTerraformGitHub ActionsGreat ExpectationsDeequIAMKMSVPC EndpointsLake FormationCloudTrailCloudWatchCI/CDIaC

Good to Have

SnowflakeApache FlinkRedshift SpectrumOpenLineageOPApolicy as code

Highlights

  • Actively hiring

More Details

RoleSenior Technical Lead (IN)
DepartmentData Engineering
Employment TypeFull Time, Remote

About the Company

HCLTech logo

HCLTech

IT Consulting

Senior Technical Lead (IN) at HCLTech | SkillMX | SkillMX