Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and shape the future of hybrid cloud and AI. You'll build long-term client relationships, collaborate globally, and drive meaningful change using robust IBM technology and Red Hat. Our culture fosters curiosity, innovation, and your long-term career development, valuing your unique contributions.
This role focuses on designing, building, and maintaining data engineering solutions within Google's Cloud ecosystem. You will leverage various Google services for both batch and real-time data pipelines, data migration, and data layer design.
As a Data Engineer, your key responsibilities will involve designing and developing robust batch and real-time data pipelines for data warehouses and data lakes. This includes utilizing Google services like DataProc, DataFlow, PubSub, BigQuery, and BigTable.
You will also develop and maintain data engineering solutions using Google Cloud Storage, BigTable, and DataProc with Spark/Hadoop, or DataFlow with Apache Beam/Python. Furthermore, you'll manage the data platform through Google Cloud Scheduler and Cloud Composer (Airflow), ensuring efficient pipeline operations. Implementing data migration solutions and optimizing the data layer with services such as BigQuery, BigTable, and Cloud Spanner are also critical aspects of this role.
We are seeking individuals with extensive experience in Google's Cloud ecosystem, including designing and building data engineering solutions. Proficiency in developing and managing batch and real-time data pipelines for data warehouses and data lakes using Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala is essential.
Demonstrated experience working with Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam/Python is required. Knowledge of scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow) is also expected. Understanding data layer design with Google services like BigQuery, BigTable, and Cloud Spanner, alongside experience in implementing data migration solutions, will be highly valued.
IBM
Software Development