Data Engineer-Data Platforms-AWS
IBM
IBM
Join IBM Consulting and shape the future of hybrid cloud and AI for leading companies worldwide. As a Data Engineer specializing in Data Platforms on AWS, you will be instrumental in designing, developing, and maintaining robust data engineering solutions within the AWS Cloud ecosystem. This role offers a unique opportunity to collaborate with clients and leverage cutting-edge IBM technology and Red Hat solutions.
At IBM Consulting, curiosity and innovation are at the core of our success. We encourage challenging the status quo, exploring novel ideas, and creating impactful solutions that drive tangible results. Our supportive culture fosters long-term career development, valuing your distinct skills and experiences as you contribute to client success.
Your primary responsibilities will include designing and building scalable batch and real-time data pipelines utilizing key AWS services such as EMR, Glue, Glue Catalog, and Kinesis. You will also create and manage data layers on AWS RedShift, Aurora, and DynamoDB, and perform data migrations using AWS DMS.
Develop sophisticated batch and real-time data pipelines for Data Warehouse and Datalake environments, leveraging AWS Kinesis and Managed Streaming for Apache Kafka. Effectively schedule and manage data services on the AWS Platform to ensure seamless operation and integration of all data engineering solutions.
Apply open-source technologies like Apache Airflow and dbt, alongside Spark with Python or Scala on the AWS Platform, to enhance and support your data engineering solutions. This role requires hands-on experience with the complete AWS toolset for data pipeline development and management.
We are seeking individuals with practical experience in developing batch and real-time data pipelines for Data Warehouse and Datalake scenarios, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. Experience in creating data layers using AWS RedShift, Aurora, and DynamoDB, and migrating data with AWS DMS is essential.
Proficiency in applying open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS Platform, is required. You should have demonstrated experience working with AWS services including EMR, Glue, Glue Catalog, and Kinesis for pipeline design and operation.
Familiarity with AWS Glue Databrew for data preparation and quality, knowledge of Lambda functions with Python for data processing, and experience with RedShift Spectrum for data warehousing and analytics are highly desirable. You will be responsible for scheduling and managing data services on the AWS Platform.
IBM
Information Technology & Services