GCP Data Engineer
Persistent
Persistent
Join our team as an experienced GCP Data Engineer and play a pivotal role in migrating a large-scale wholesale banking data mart from on-premises Teradata to Google BigQuery. You'll leverage your deep expertise in Google Cloud Platform, BigQuery, and Python to design, build, and optimize data solutions.
This role offers the chance to work with cutting-edge cloud-native data engineering practices, ensuring secure and compliant data migration while delivering high-performance analytics. We are an AI-led digital engineering partner committed to client success and innovation, with a global presence across 21 countries.
Design and implement scalable batch and event-driven data pipelines on Google Cloud Platform, focusing on building enterprise data warehouses and data marts within BigQuery.
Lead critical migration activities from Teradata to BigQuery, encompassing schema conversion, SQL transformations, and rigorous validation.
Develop and maintain robust workflow orchestration using Cloud Composer (Apache Airflow) and implement event-driven architectures with Pub/Sub and Cloud Functions.
Create efficient data ingestion frameworks, optimize BigQuery workloads through partitioning and clustering, and design data models for advanced analytics and reporting.
Develop Python utilities for automation, manage GCP resources securely with IAM, and implement comprehensive operational monitoring and alerting.
A Bachelor's Degree in Computer Science, IT, Engineering, MCA, or BCA is required, alongside 5-10 years of dedicated experience in Data Engineering or Cloud Platform Engineering.
Demonstrate a minimum of 3+ years of hands-on experience with Google Cloud Platform (GCP) and proven success in building enterprise data platforms utilizing BigQuery.
Possess strong proficiency in Python programming and advanced SQL for complex query development and optimization.
Familiarity with building and supporting data pipelines using Cloud Composer (Apache Airflow) and implementing event-driven architectures with Pub/Sub and Cloud Functions is essential.
Experience with Teradata data migration projects and a solid understanding of data warehousing concepts, ETL/ELT frameworks, and data integration are highly valued.
Persistent Systems
IT Consulting