Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and build lasting client relationships while shaping the future of hybrid cloud and AI. This role offers the opportunity to work with leading companies across various industries, leveraging IBM technology, Red Hat, and support from strategic partners to drive meaningful change. We foster a culture of curiosity, innovation, and continuous growth, valuing your unique contributions and supporting your long-term career development.
As a Data Engineer focused on Google's data platforms, you will be instrumental in designing, developing, and maintaining robust data engineering solutions within the Google Cloud ecosystem. This position demands expertise in creating both batch and real-time data pipelines, executing data migrations, and architecting efficient data layers.
Key responsibilities include designing and developing comprehensive batch and real-time data pipelines for data warehouses and data lakes utilizing Google services like DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. You will build and maintain data engineering solutions using Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam or Python.
Furthermore, you will manage data platform operations by scheduling and overseeing processes with Google Cloud Scheduler and Cloud Composer (Airflow). This role also involves developing and implementing effective data migration strategies and optimizing the data layer using services such as BigQuery, BigTable, and Cloud Spanner for enhanced data storage and retrieval.
This role requires hands-on experience in designing, building, and maintaining data engineering solutions within Google's Cloud ecosystem, including services such as Google DataProc, DataFlow, Pub/Sub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB.
You should possess experience in developing and managing batch and real-time data pipelines for data warehouses and data lakes using Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala. Proficiency in Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam or Python is essential. Familiarity with scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow) is also expected.
IBM
Technology