Data Engineer-Data Platforms-Google
IBM
IBM
Embark on a career shaping hybrid cloud and AI journeys for leading companies. This role focuses on designing, building, and maintaining robust data engineering solutions within Google's Cloud ecosystem. You will leverage cutting-edge Google services to create efficient batch and real-time data pipelines, manage data migrations, and optimize data layer architecture.
Key responsibilities include designing and developing batch and real-time data pipelines using services like DataProc, DataFlow, PubSub, and BigQuery. You will build and maintain data engineering solutions leveraging Google Cloud Storage, BigTable, and DataFlow with Apache Beam or Python. Managing data platforms through Cloud Scheduler and Cloud Composer (Airflow) is crucial, as is implementing seamless data migration solutions. Optimizing the data layer with BigQuery, BigTable, and Cloud Spanner ensures efficient data storage and retrieval.
This role requires hands-on experience with Google's Cloud ecosystem, including services such as DataProc, DataFlow, PubSub, BigQuery, and BigTable. Proficiency in developing and managing data pipelines with technologies like Spark/Python and Apache Airflow is essential. Experience with data layer design and implementing data migration solutions using Google services is expected. Familiarity with open-source technologies like dbt and Cloud Composer (Airflow) for platform management is a significant advantage.
IBM
Software