Data Product Manager (IN)
HCLTech
HCLTech
Lead the design and scaling of our next-generation data platform as a Senior Databricks Data Engineer. You will serve as the technical authority for our Databricks on AWS environment, architecting efficient pipelines for massive datasets that drive our analytics and AI efforts. This role involves native integration with AWS services, stringent governance, and performance optimization.
We are seeking an experienced engineer to architect and implement robust, cost-effective data pipelines. This position offers a unique opportunity to shape our data infrastructure and leverage cutting-edge technologies.
Architect and build end-to-end data products utilizing the Medallion Architecture (Bronze, Silver, Gold) on Amazon S3 and Delta Lake. Design and scale production pipelines with PySpark, SQL, and Delta Live Tables for batch and real-time streaming data. Integrate Databricks with AWS messaging systems like Kinesis or MSK for diverse data ingestion. Optimize Lakehouse storage using Delta features like Liquid Clustering and Z-Ordering. Implement data governance and role-based access control via Unity Catalog. Manage infrastructure security and optimize cloud spend by configuring Databricks clusters and EC2 instances. Develop and deploy Infrastructure as Code using Terraform and establish CI/CD pipelines. Mentor mid-level engineers and enforce coding standards through peer reviews.
Requires a minimum of 6 years in Data Engineering, with at least 4 years focused on Databricks within an AWS environment. Expertise in Python (PySpark) and complex analytical SQL is essential. Deep technical knowledge of Delta Lake, Amazon S3, and advanced Spark optimization techniques is required. Strong hands-on experience with AWS services including EC2, IAM, S3, Secrets Manager, and VPC networking fundamentals is expected. Proven experience with Terraform for data platform provisioning and automated deployment tools (CI/CD) is necessary. Excellent architectural ownership and technical communication skills, including the ability to lead design discussions and document complex data lineage, are vital.
HCLTech
IT Consulting