Data Engineer-Data Platforms-AWS
IBM
IBM
Join IBM Consulting and be part of a dynamic team shaping the future of hybrid cloud and AI for leading companies worldwide. We foster a culture of curiosity and innovation, empowering you to challenge norms and create impactful solutions. Your long-term career development is a priority, valuing your unique contributions.
As a Data Engineer specializing in Data Platforms on AWS, you will play a key role in designing, developing, and maintaining robust data engineering solutions within the AWS Cloud ecosystem. This position requires expertise in building and operating both batch and real-time data pipelines, ensuring the seamless flow and integration of data across various AWS services.
Your core responsibilities will involve designing and building scalable batch and real-time data pipelines utilizing AWS services such as EMR, Glue, Glue Catalog, and Kinesis. You'll be instrumental in creating essential data layers on AWS RedShift, Aurora, and DynamoDB, including data migration using AWS DMS.
Further, you will manage and schedule data services on the AWS platform, ensuring optimal performance. This includes developing pipelines for Data Warehouses and Data Lakes using AWS Kinesis and MSK, and leveraging open-source technologies like Apache Airflow and dbt. Proficiency with Spark/Python or Spark/Scala on AWS is essential.
We are looking for candidates with hands-on experience in the AWS toolset, specifically with EMR, Glue, Glue Catalog, and Kinesis for pipeline development. Experience in creating data layers with RedShift, Aurora, and DynamoDB, and data migration via AWS DMS is crucial.
Familiarity with open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on AWS, is required. You should also have experience in managing data services, working with AWS Glue Databrew for data preparation, and utilizing Lambda functions with Python for data processing and transformation.
Knowledge of RedShift Spectrum for data warehousing and analytics tasks is beneficial.
IBM
Data Analytics