Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting to collaborate with leading companies worldwide, guiding them through their hybrid cloud and AI transformations. Leveraging IBM technology, Red Hat, and strategic partnerships, you'll drive significant change and deliver impactful client results. Our culture fosters curiosity, encourages challenging the status quo, and embraces innovation to achieve tangible outcomes. We prioritize your long-term career growth and value your unique contributions.
As a Data Engineer specializing in Google's data platforms, you will architect, build, and manage robust data engineering solutions within the Google Cloud ecosystem. This position demands proficiency in leveraging various Google services for both batch and real-time data pipelines, alongside expertise in data migration and data layer design.
Key responsibilities include designing and developing scalable batch and real-time data pipelines for Data Warehouses and Data Lakes using Google services like DataProc, DataFlow, PubSub, BigQuery, and BigTable.
You will build and maintain data engineering solutions utilizing Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python. Effective scheduling and management of the data platform using Google Cloud Scheduler and Cloud Composer (Airflow) are crucial for ensuring efficient operations.
This role also involves developing and implementing comprehensive data migration solutions and optimizing the data layer using Google services such as BigQuery, BigTable, and Cloud Spanner to ensure efficient data storage and retrieval.
Demonstrate expertise in designing, building, and maintaining data engineering solutions on Google Cloud, encompassing services like DataProc, DataFlow, PubSub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB.
Possess hands-on experience developing and managing batch and real-time data pipelines for Data Warehouses and Data Lakes, utilizing Google services alongside open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala.
Proficiency is required with Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python. Experience in data layer design using BigQuery, BigTable, and Cloud Spanner is also essential. Familiarity with Cloud Composer (Airflow) for platform scheduling and management, and implementing data migration solutions using Google services are key qualifications.
IBM
Information Technology & Services