Data Engineer-Data Platforms-AWS
IBM
IBM
Join IBM Consulting as a Data Engineer focused on Data Platforms within the AWS ecosystem. This role involves collaborating with leading companies to shape their hybrid cloud and AI strategies. You will develop and maintain robust data engineering solutions, leveraging IBM's technology and strategic partnerships to drive significant client impact.
At IBM Consulting, curiosity and innovation are highly valued. We encourage challenging the status quo and exploring new ideas to create effective solutions. Our supportive culture prioritizes long-term career development, recognizing and celebrating your unique skills and contributions.
Design, build, and manage batch and real-time data pipelines utilizing AWS services like EMR, Glue, and Kinesis. Develop data layers on RedShift, Aurora, and DynamoDB, and perform data migration using AWS DMS. Schedule and manage data services efficiently on the AWS platform. Create comprehensive data pipelines for Data Warehouses and Data Lakes, incorporating AWS Kinesis and Managed Streaming for Apache Kafka.
Utilize open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS platform. Gain experience with AWS Glue Databrew for data preparation and quality. Apply Lambda functions with Python for data processing and transformation. Work with RedShift Spectrum for advanced data warehousing and analytics tasks.
Proven experience in designing and developing batch and real-time data pipelines on AWS. Expertise in AWS services including EMR, Glue, Glue Catalog, Kinesis, RedShift, Aurora, and DynamoDB. Familiarity with data migration tools like AWS DMS. Proficiency in using open-source technologies such as Apache Airflow and dbt. Experience with Spark/Python or Spark/Scala. Exposure to AWS Lambda functions and RedShift Spectrum is beneficial.
Understanding of data warehousing, data lakes, data preparation, and data quality concepts is essential. The ability to schedule and manage data services on the AWS platform is required. Experience with AWS Glue Databrew and Managed Streaming for Apache Kafka is a plus.
IBM
Information Technology & Services