Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and contribute to long-term client relationships and global collaboration, shaping hybrid cloud and AI journeys. You'll leverage IBM technology, Red Hat, and strategic partnerships to drive significant change and client impact. Our culture fosters curiosity, innovation, and continuous growth, prioritizing your career development and valuing your unique contributions.
This role focuses on designing, building, and maintaining robust data engineering solutions within Google's Cloud ecosystem. You will specialize in developing both batch and real-time data pipelines, implementing data migration strategies, and architecting efficient data layers using a variety of Google services.
Design and develop both batch and real-time data pipelines for Data Warehouses and Datalakes. Utilize Google services like DataProc, DataFlow, PubSub, BigQuery, and BigTable. Build and maintain data engineering solutions using Google Cloud Storage, BigTable, BigQuery DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python). Schedule and manage data platform operations efficiently using Google Cloud Scheduler and Cloud Composer (Airflow).
Implement comprehensive data migration solutions to ensure seamless data transfer. Optimize data layers using services such as BigQuery, Big Table, and Cloud Spanner for efficient storage and retrieval. Collaborate to ensure smooth data pipeline operations and effective data management.
Demonstrated expertise in designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem. Proven experience developing and managing batch and real-time data pipelines using Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala. Proficiency in Google Cloud Storage, BigTable, BigQuery DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python).
Knowledge of scheduling and managing data platforms via Google Cloud Scheduler and Cloud Composer (Airflow). Understanding of data layer design with BigQuery, Big Table, and Cloud Spanner. Experience with data migration solutions using Google services. Familiarity with open-source tools such as Apache Airflow, dbt, Spark/Python, and Spark/Scala is essential for success in this role.
IBM
IT Consulting