Data Engineer-Data Platforms-Google
IBM
IBM
Join IBM Consulting and contribute to long-term client relationships by shaping hybrid cloud and AI journeys for leading companies. Leverage robust IBM technology and Red Hat with our strategic partners to drive significant change and accelerate client impact. We foster a culture of curiosity, innovation, and continuous growth, supporting your career development while valuing your unique contributions.
As a Data Engineer on Google's data platforms, you will design, build, and maintain robust data engineering solutions within the Google Cloud ecosystem. This includes developing batch and real-time data pipelines, managing data migration, and architecting data layers using services like DataProc, DataFlow, PubSub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB. You will also schedule and manage the data platform, ensuring seamless workflows and pipeline operation.
Key responsibilities involve designing and implementing data pipelines for Data Warehouse and Data Lakes, optimizing them for efficiency and scalability. You'll work with Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python. Familiarity with scheduling tools like Google Cloud Scheduler and Cloud Composer (Airflow) is essential for managing the data platform effectively.
We are seeking candidates with a strong understanding of designing, building, and maintaining data engineering solutions on Google's Cloud ecosystem. Experience with developing and managing batch and real-time data pipelines for Data Warehouse and Data Lakes, including data migration and integration, is required. Proficiency in utilizing Google services such as DataProc, DataFlow, PubSub, BigQuery, BigTable, Cloud Spanner, CloudSQL, and AlloyDB is crucial.
Familiarity with open-source technologies like Spark/Python, Apache Airflow, dbt, and Spark/Scala is highly advantageous. Experience with data platform management using Google Cloud Scheduler and Cloud Composer (Airflow) is also beneficial. This role requires a proactive approach to optimizing data pipelines for performance, scalability, and reliability, ensuring high-quality data delivery.
IBM
Data Analytics