Senior Technical Lead (IN)
HCLTech
HCLTech
Join our team as a Core Data Engineer specializing in AWS. You will be instrumental in designing, building, and maintaining secure, reliable, and cost-effective data pipelines. This role involves handling batch, streaming, and Change Data Capture (CDC) ingestion, culminating in curated datasets prepared for advanced analytics. A strong emphasis will be placed on integrating testing, governance, observability, and CI/CD practices throughout the data lifecycle.
Key responsibilities include implementing robust ingestion pipelines using AWS Lambda and EMR for batch and streaming data, managing schema evolution, and orchestrating workflows with Step Functions or MWAA (Airflow). You will model datasets on S3 using formats like Parquet and Iceberg, manage data quality with automated tests and CI gates, and implement comprehensive security and governance measures including IAM, KMS, and Lake Formation policies. Driving observability through CloudWatch and optimizing costs via FinOps principles are also crucial. This role requires provisioning infrastructure with IaC tools like Terraform and building CI/CD pipelines with GitHub Actions, alongside maintaining essential documentation and runbooks.
We are seeking candidates with over 10 years of experience in building data platforms on AWS, demonstrating strong proficiency in Python (including PySpark) and SQL. Hands-on experience with AWS services such as DMS, Kinesis/MSK, Glue/EMR, Lambda, Step Functions/MWAA, S3/Parquet/Iceberg, Glue Data Catalog, Athena, and Redshift/Serverless is essential. Expertise in Infrastructure as Code (Terraform) and CI/CD (GitHub Actions) is required, along with experience in data quality frameworks like Great Expectations/Deequ, schema evolution, and security/compliance (IAM/KMS, private networking, Lake Formation, audit/lineage). A proven ability to tune performance and costs across storage and compute is also necessary.
HCLTech
IT Consulting