Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and collaborate with leading companies globally to shape their hybrid cloud and AI strategies. You will leverage IBM technology, Red Hat, and support from strategic partners to drive impactful change and accelerate client success. We foster a culture of curiosity, innovation, and growth, empowering you to challenge the status quo and develop your long-term career.
As a Data Engineer, you will focus on designing, building, and maintaining robust data engineering solutions within Google's Cloud ecosystem. This role demands expertise in Google services for both batch and real-time data pipelines, facilitating data migration, and crafting efficient data layer designs.
Your core responsibilities will involve architecting and developing comprehensive data pipelines, including batch and real-time solutions for data warehouses and data lakes. You will utilize key Google services like DataProc, DataFlow, PubSub, BigQuery, and Big Table. Furthermore, you will build and maintain data engineering solutions using Google Cloud Storage, BigTable, DataProc with Spark/Hadoop, and DataFlow with Apache Beam or Python.
Efficiently manage data platform operations through Google Cloud Scheduler and Cloud Composer (Airflow). Implement seamless data migration strategies using Google services. Optimize data layers with services such as BigQuery, Big Table, and Cloud Spanner for enhanced storage and retrieval. Leverage open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala to enhance pipeline development and management.
We are seeking candidates with extensive experience in designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem. Proficiency in Google services including DataProc, DataFlow, PubSub, BigQuery, Big Table, Cloud Spanner, CloudSQL, and AlloyDB is essential.
Demonstrated experience in developing and managing batch and real-time data pipelines for Data Warehouse and Datalake using Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala is crucial. You should have hands-on experience with Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam or Python. Familiarity with scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow) is expected. A solid understanding of data layer design using BigQuery, Big Table, and Cloud Spanner is required, alongside experience in implementing data migration solutions.
IBM
IT Consulting