Data Engineer-Data Platforms-Google
IBM
IBM
Embark on a transformative career with IBM Consulting, where we build enduring client partnerships through global collaboration. You will collaborate with industry leaders, guiding them through their hybrid cloud and AI journeys. Empowered by our strategic alliances, robust IBM technology, and Red Hat, you'll possess the tools to drive substantial change and accelerate client success.
Curiosity is the engine of achievement at IBM Consulting. We encourage you to question the status quo, explore novel concepts, and craft innovative solutions that yield tangible results. Our culture promotes growth and empathy, prioritizing your long-term career development while celebrating your distinct skills and experiences.
As a Data Engineer focused on Google's data platforms, you will be instrumental in designing, constructing, and maintaining sophisticated data engineering solutions within Google's Cloud ecosystem. This role demands proficiency in leveraging various Google services for both batch and real-time data pipelines, seamless data migration, and effective data layer design.
Key responsibilities include: - Architecting and developing robust batch and real-time data pipelines for Data Warehouses and Data Lakes using Google services such as DataProc, DataFlow, PubSub, BigQuery, and Bigtable. - Building and sustaining data engineering solutions by utilizing Google Cloud Storage, Bigtable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python. - Ensuring efficient data pipeline operations through scheduling and managing the data platform with Google Cloud Scheduler and Cloud Composer (Airflow). - Implementing comprehensive data migration solutions using Google services for smooth data transfer across systems. - Designing and optimizing the data layer with Google services like BigQuery, Bigtable, and Cloud Spanner for high-performance data storage and retrieval.
This role requires hands-on experience in designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem, encompassing services like Google DataProc, DataFlow, PubSub, BigQuery, Bigtable, Cloud Spanner, CloudSQL, and AlloyDB.
Demonstrated experience in developing and managing batch and real-time data pipelines for Data Warehouses and Data Lakes using Google services and open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala is essential.
Proficiency with Google Cloud Storage, Bigtable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python is expected. Familiarity with scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow) for efficient operations is also required.
Candidates should possess a solid understanding of data layer design using Google services such as BigQuery, Bigtable, and Cloud Spanner for efficient data storage and retrieval. Experience with open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala for pipeline development and management is crucial. Experience in developing and implementing data migration solutions using Google services is necessary.
IBM
Software Development