Data Engineer-Data Platforms-Google
IBM
IBM
Embark on a career within IBM Consulting, where long-term client relationships and global collaboration are foundational. You will partner with leading organizations across diverse industries, guiding them through their hybrid cloud and AI transformations. Leveraging robust IBM technology, Red Hat, and support from our strategic partners, you'll possess the essential tools to drive significant change and amplify client impact.
At IBM Consulting, curiosity is the catalyst for success. We foster an environment that encourages challenging the status quo, exploring novel concepts, and developing innovative solutions that yield tangible results. Our culture emphasizes growth and empathy, focusing on your long-term career trajectory while celebrating your unique skills and experiences.
As a Data Engineer specializing in Google's data platforms, you will be instrumental in designing, constructing, and maintaining robust data engineering solutions within the Google Cloud ecosystem. This role demands proficiency in leveraging various Google services for both batch and real-time data pipelines, alongside expertise in data migration strategies and data layer architecture.
Key responsibilities include: - Designing and developing batch and real-time data pipelines for Data Warehouses and Data Lakes utilizing Google services like DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. - Building and maintaining data engineering solutions with Google Cloud Storage, BigTable, BigQuery DataProc (Spark/Hadoop), and Google DataFlow (Apache Beam/Python). - Managing the data platform through Google Cloud Scheduler and Cloud Composer (Airflow) to ensure streamlined data pipeline operations. - Developing and implementing efficient data migration solutions using Google services. - Optimizing the data layer architecture with Google services such as BigQuery, BigTable, and Cloud Spanner for enhanced data storage and retrieval.
To excel in this role, you will need extensive exposure to designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem. This includes hands-on experience with services such as Google DataProc, DataFlow, Pub/Sub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB.
Demonstrated experience in developing and managing batch and real-time data pipelines for Data Warehouses and Data Lakes is crucial. Proficiency with Google services and open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala is required. You should possess strong skills in working with Google Cloud Storage, BigTable, BigQuery DataProc, and Google DataFlow.
Familiarity with scheduling and managing data platforms using Google Cloud Scheduler and Cloud Composer (Airflow) is essential. Understanding data layer design principles, particularly with BigQuery, BigTable, and Cloud Spanner, for efficient data management is key. Experience in developing and implementing data migration solutions using Google services is also vital.
IBM
IT Consulting