Data Engineer-Data Platforms-AWS
IBM
IBM
Embark on a career with IBM Consulting, where you'll build lasting client relationships and collaborate globally. You will empower leading companies across diverse industries in their hybrid cloud and AI transformation. Leveraging strategic partnerships, robust IBM technology, and Red Hat, you'll be equipped to drive significant change and accelerate client success. At IBM Consulting, a curious mindset is key. We encourage challenging the status quo, exploring novel ideas, and crafting innovative solutions that deliver tangible results. Our growth-oriented and empathetic culture fosters your long-term career development while valuing your unique contributions.
As a Data Engineer focused on Data Platforms within the AWS ecosystem, you will advise on, develop, and maintain data engineering solutions. This includes designing, building, and operating batch and real-time data pipelines using various AWS services. Key responsibilities involve creating data layers on AWS RedShift, Aurora, and DynamoDB, migrating data with AWS DMS, and managing data services on the AWS Platform.
You will develop batch and real-time pipelines for Data Warehouses and Data Lakes, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. The role requires leveraging open-source technologies like Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS Platform.
This role requires hands-on experience with AWS services including EMR, Glue, Glue Catalog, and Kinesis for designing and operating data pipelines. You should have experience with AWS RedShift, Aurora, and DynamoDB for data layer creation and data migration using AWS DMS.
Familiarity with open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS Platform is essential. Experience in scheduling and managing data services on AWS, proficiency with AWS Glue Databrew for data preparation and quality, and knowledge of Lambda functions with Python for data processing are expected.
An understanding of RedShift Spectrum for data warehousing and analytics tasks is also beneficial for this position.
IBM
Information Technology & Services