Data Engineer-Data Platforms-AWS
IBM
IBM
Join IBM Consulting and become a key player in shaping the future of hybrid cloud and AI for leading companies worldwide. This role focuses on designing, developing, and maintaining robust data engineering solutions within the AWS Cloud ecosystem. You will be instrumental in building and operating efficient batch and real-time data pipelines.
At IBM Consulting, we foster a culture of curiosity and innovation. We empower you to challenge the status quo, explore groundbreaking ideas, and develop inventive solutions that drive tangible client impact. Your career growth and unique contributions are highly valued in our supportive environment.
Your core responsibilities will involve designing, building, and operating scalable batch and real-time data pipelines using essential AWS services like EMR, Glue, Glue Catalog, and Kinesis. You'll create vital data layers on RedShift, Aurora, and DynamoDB, facilitating data migration with AWS DMS. Furthermore, you will manage and schedule data services on the AWS Platform for optimal performance, and develop data pipelines for Data Warehouse and Data Lakes, leveraging AWS Kinesis and MSK.
You will also utilize popular open-source technologies such as Apache Airflow and dbt, alongside Spark with Python or Scala, to enhance our data engineering solutions. This includes working with AWS Databrew for data preparation and quality, Lambda functions for processing and transformation, and RedShift Spectrum for warehousing and analytics.
We are seeking individuals with proven experience in developing batch and real-time data pipelines for Data Warehousing and Data Lakes, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. Proficiency in creating data layers on AWS RedShift, Aurora, and DynamoDB, and migrating data via AWS DMS is essential.
Your background should include experience with open-source technologies like Apache Airflow and dbt, and Spark (Python or Scala) on the AWS Platform. Familiarity with AWS Databrew, Lambda functions (Python), and RedShift Spectrum is also crucial for this role. Effective scheduling and management of data services on AWS are key competencies required.
IBM
Information Technology & Services