Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and build influential client relationships worldwide. You will collaborate with industry leaders to drive their hybrid cloud and AI transformation. Leveraging IBM's technology, Red Hat, and strategic partnerships, you'll innovate and deliver impactful solutions.
Curiosity is key to success at IBM Consulting. We encourage challenging the status quo, exploring new concepts, and crafting groundbreaking solutions that yield tangible outcomes. Our supportive and growth-oriented culture champions your career development while celebrating your unique contributions.
As a Data Engineer focused on Google's data platforms, you will architect, develop, and maintain robust data engineering solutions within the Google Cloud ecosystem.
Key responsibilities include: - Designing and developing batch and real-time data pipelines for data warehouses and data lakes using Google services like DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. - Building and managing data engineering solutions with Google Cloud Storage, BigTable, BigQuery DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python). - Scheduling and managing the data platform via Google Cloud Scheduler and Cloud Composer (Airflow) for optimized pipeline operations. - Implementing comprehensive data migration solutions using Google services for seamless data transfers. - Designing and optimizing the data layer with services such as BigQuery, BigTable, and Cloud Spanner for efficient data storage and retrieval.
To excel in this role, you'll need hands-on experience with Google's Cloud ecosystem for data engineering. This includes designing, building, and maintaining solutions using services like DataProc, DataFlow, Pub/Sub, BigQuery, BigTable, Cloud Spanner, Cloud SQL, and AlloyDB.
Proficiency in developing and managing batch and real-time data pipelines for data warehouses and data lakes is essential. Familiarity with open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala is highly valued.
We seek experience in managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow), along with expertise in designing data layers with BigQuery, BigTable, and Cloud Spanner. Experience implementing data migration solutions using Google services is also required.
IBM
Information Technology & Services