Data Engineer-Data Platforms-Google

IBM

14–16 yrs Bengaluru Full Time Work from office
IBM logo
Posted : yesterday
Actively hiring

Job description

IBM Consulting is seeking a skilled Data Engineer to join our FutureNow Centers, contributing to client success in hybrid cloud and AI transformations. This role focuses on leveraging Google's data platforms to design, build, and maintain robust data engineering solutions. You will collaborate with leading companies across various industries, driving innovation and delivering impactful results. Our culture fosters curiosity, innovation, and continuous learning, supporting your long-term career development and valuing your unique contributions.

Responsibilities

Design and develop scalable batch and real-time data pipelines for data warehouses and data lakes using Google Cloud services like DataProc, DataFlow, Pub/Sub, BigQuery, and BigTable. Develop comprehensive data engineering solutions utilizing Google Cloud Storage, BigTable, BigQuery, DataProc (Spark/Hadoop), and DataFlow (Apache Beam/Python). Manage and schedule data platform operations using Google Cloud Scheduler and Cloud Composer (Airflow) for optimal pipeline efficiency. Implement effective data migration strategies using Google services to ensure seamless data transfer between systems. Design and optimize data layers with Google services such as BigQuery, BigTable, and Cloud Spanner for efficient data storage and retrieval.

Qualifications

Proven expertise in designing, building, and maintaining data engineering solutions within the Google Cloud ecosystem, including services like DataProc, DataFlow, Pub/Sub, BigQuery, BigTable, Cloud Spanner, Cloud SQL, and AlloyDB. Demonstrated experience in developing and managing batch and real-time data pipelines for Data Warehouses and Data Lakes, utilizing Google services and open-source technologies such as Apache Airflow, dbt, Spark/Python, or Spark/Scala. Proficiency in working with Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python for building data engineering solutions. Solid understanding of data platform management, including scheduling and managing operations with Google Cloud Scheduler and Cloud Composer (Airflow). Experience in data layer design and optimization using Google services like BigQuery, BigTable, and Cloud Spanner. Bachelor's Degree is required; a Master's Degree is preferred.

Essential Skills

Google Cloud PlatformData PipelinesData WarehousingData LakesBatch ProcessingReal-time ProcessingData MigrationData Layer DesignGoogle DataProcGoogle DataFlowGoogle Pub/SubGoogle BigQueryGoogle BigTableGoogle Cloud StorageGoogle Cloud SchedulerGoogle Cloud ComposerApache AirflowApache BeamPythonSparkHadoopGoogle Cloud SpannerGoogle Cloud SQLAlloyDB

Good to Have

dbtScala

Highlights

  • Actively hiring

More Details

RoleData Engineer-Data Platforms-Google
IndustryManagement Consulting
DepartmentData Engineering
Employment TypeFull Time, Work from office

About the Company

IBM logo

IBM

Management Consulting

Data Engineer-Data Platforms-Google at IBM | SkillMX | SkillMX