Data Engineer-Data Platforms-Google
IBM
IBM
Shape the future of hybrid cloud and AI journeys with IBM Consulting. This role focuses on designing, building, and maintaining robust data engineering solutions within Google's comprehensive cloud ecosystem. You'll leverage cutting-edge Google services to create efficient batch and real-time data pipelines, manage data migrations, and optimize data layer designs for leading companies.
We foster a culture of curiosity and continuous growth, encouraging you to explore new ideas and develop innovative solutions. Your long-term career development is a priority, and your unique skills and experiences are highly valued as we collaborate worldwide to drive significant client impact.
As a Data Engineer specializing in Google's data platforms, your key responsibilities will include:
- Designing and developing scalable batch and real-time data pipelines using Google services like DataProc, DataFlow, PubSub, BigQuery, and BigTable. - Building and maintaining sophisticated data engineering solutions with Google Cloud Storage, BigTable, BigQuery, DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python). - Scheduling and managing data platform operations through Google Cloud Scheduler and Cloud Composer (Airflow) for seamless pipeline execution. - Implementing effective data migration strategies and solutions across various systems. - Designing and optimizing data layers using BigQuery, BigTable, and Cloud Spanner for efficient data management.
We are seeking individuals with extensive experience in:
- Designing, building, and maintaining data engineering solutions on Google Cloud, utilizing services such as DataProc, DataFlow, PubSub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB. - Developing and managing batch and real-time data pipelines for Data Warehouses and Datelakes with Google services and open-source tools like Apache Airflow, dbt, Spark/Python, or Spark/Scala. - Proficiently working with Google Cloud Storage, BigTable, BigQuery, DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python). - Understanding data platform management, including scheduling and orchestration with Google Cloud Scheduler and Cloud Composer (Airflow). - Experience in data layer design using BigQuery, BigTable, and Cloud Spanner. - Utilizing open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala for data pipeline development. - Developing and implementing data migration solutions using Google services. - Expertise with Cloud Composer (Airflow) for platform management.
IBM
IT Consulting