Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and shape the future of hybrid cloud and AI for leading companies worldwide. We foster long-term client relationships through close collaboration, leveraging IBM technology, Red Hat, and strategic partnerships. Curiosity and innovation drive our success, as we encourage exploring new ideas to deliver impactful solutions. Our growth-focused culture supports your career development, valuing your unique contributions.
As a Data Engineer, you will be instrumental in designing, building, and maintaining robust data engineering solutions within Google's Cloud ecosystem. This role emphasizes expertise in developing both batch and real-time data pipelines, managing data migration, and architecting efficient data layers using a variety of Google services.
Key responsibilities include architecting and developing scalable batch and real-time data pipelines for data warehouses and data lakes using Google services like DataProc, DataFlow, PubSub, BigQuery, and BigTable.
You will also develop and maintain data engineering solutions leveraging Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam or Python. Efficient data platform operations will be ensured through scheduling and management using Google Cloud Scheduler and Cloud Composer (Airflow).
Furthermore, this role involves implementing comprehensive data migration strategies and designing/optimizing the data layer using services such as BigQuery, BigTable, and Cloud Spanner for enhanced data storage and retrieval.
Essential expertise includes designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem, encompassing services like Google DataProc, DataFlow, PubSub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB.
Demonstrated experience in developing and managing batch and real-time data pipelines for Data Warehouse and Datalake environments is crucial. Proficiency in using Google Cloud Storage, BigTable, BigQuery DataProc with Spark/Hadoop, and Google DataFlow with Apache Beam or Python is expected.
Familiarity with scheduling and managing data platforms via Google Cloud Scheduler and Cloud Composer (Airflow) is beneficial. Experience with data layer design using BigQuery, BigTable, and Cloud Spanner, alongside open-source technologies like Apache Airflow, dbt, Spark/Python, or Spark/Scala, will be highly valued. Experience in developing and implementing data migration solutions using Google services is also a key requirement.
IBM
Management Consulting