Technical Specialist (IN)
HCLTech
HCLTech
Join our team as a Senior GCP Data Engineer and play a pivotal role in building and optimizing scalable data platforms on Google Cloud Platform. We are looking for an experienced professional with a strong command of Python, PySpark, and SQL to transform complex data into reliable, secure, and insightful assets. This position demands technical leadership and a commitment to production engineering discipline, enabling the creation of resilient cloud data solutions that meet evolving business needs.
This role focuses on designing, constructing, and maintaining robust data pipelines and platforms. You will be instrumental in developing advanced ETL/ELT processes, modeling data effectively, and ensuring optimal performance and cost efficiency within the GCP ecosystem. A key aspect involves implementing comprehensive data quality, security, and governance measures to ensure data integrity and compliance.
Architect, develop, and maintain scalable data pipelines and platforms on GCP for batch and near-real-time processing. Build robust ETL/ELT solutions using Python, PySpark, SQL, and various GCP services like BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, and Cloud Composer. Design diverse data models (dimensional, normalized, denormalized, lakehouse) to support analytical and operational goals. Develop advanced SQL transformations, stored procedures, and reusable data components. Optimize Spark jobs, BigQuery queries, and resource utilization for peak performance and cost savings. Implement data ingestion from various sources, manage schema evolution, and establish automated data quality controls, lineage, and observability. Ensure data security and governance through GCP IAM, encryption, and best practices. Develop and integrate automated testing within CI/CD pipelines and utilize infrastructure-as-code for environment consistency. Troubleshoot production issues, perform root-cause analysis, and implement corrective actions. Collaborate with cross-functional teams to translate requirements into maintainable solutions, and lead technical design reviews while mentoring junior team members.
A minimum of 8 years of professional experience in data engineering, data warehousing, big data, or cloud-based data platform development is required. Demonstrable production experience delivering data solutions on Google Cloud Platform is essential. Advanced proficiency in Python for data processing, automation, and framework development, alongside advanced skills in PySpark and Apache Spark for large-scale distributed processing and performance tuning, are critical. Expert-level SQL skills, including complex joins, window functions, and query optimization, are expected. Hands-on experience with key GCP data services such as BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, and Cloud Composer or Apache Airflow is necessary. A strong understanding of ETL/ELT patterns, batch/streaming architectures, data lakes, data warehouses, lakehouses, and orchestration is vital. Familiarity with Git, automated testing, CI/CD, logging, monitoring, incident management, and production support is required. Working knowledge of GCP IAM, networking, encryption, data governance, and secure data architecture is important. Strong analytical, problem-solving, communication, documentation, and technical leadership skills are essential. A Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field, or equivalent practical experience, will satisfy the educational requirements.
HCLTech
IT Consulting