Data Engineer-Data Platforms-Google
IBM
IBM
Embark on a career shaping hybrid cloud and AI journeys with leading companies. This role focuses on designing, building, and maintaining robust data engineering solutions within Google's comprehensive cloud ecosystem. You will leverage a wide array of Google services to create efficient batch and real-time data pipelines, manage data migrations, and architect sophisticated data layers.
Curiosity and innovation are at the heart of this consulting role. Challenge conventional approaches, explore novel ideas, and develop solutions that drive tangible client success. The culture emphasizes continuous growth and empathy, supporting your long-term career development while valuing your unique contributions.
Design and develop scalable batch and real-time data pipelines using Google services such as DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. Build and maintain data engineering solutions leveraging Google Cloud Storage, BigTable, BigQuery, and DataFlow with Apache Beam or Python. Manage and schedule data platform operations using Google Cloud Scheduler and Cloud Composer (Airflow) to ensure optimal pipeline performance.
Implement effective data migration strategies using Google services for seamless data transfer. Optimize data layer design with BigQuery, BigTable, and Cloud Spanner for efficient data storage and retrieval. Collaborate to ensure data integrity and accessibility across various platforms.
Demonstrated experience in designing, building, and maintaining data engineering solutions on Google Cloud Platform, utilizing services like DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. Proven ability to develop and manage batch and real-time data pipelines for data warehouses and data lakes using Google services and open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala.
Proficiency in working with Google Cloud Storage, BigTable, BigQuery, and Google DataFlow with Apache Beam or Python. Understanding of data platform management, including scheduling and orchestration using Google Cloud Scheduler and Cloud Composer (Airflow). Experience with data layer design using BigQuery, BigTable, and Cloud Spanner is essential. Familiarity with open-source technologies like Apache Airflow, dbt, Spark/Python, and Spark/Scala is highly valued.
IBM
Information Technology & Services