Data Engineer-Data Platforms-AWS
IBM
IBM
Join IBM Consulting and shape the future of hybrid cloud and AI. As a Data Engineer specializing in Data Platforms on AWS, you will design, develop, and maintain robust data engineering solutions within the AWS ecosystem. This role offers the opportunity to work with leading companies, driving meaningful change through innovative solutions and a culture that fosters curiosity and career growth.
Curiosity fuels success at IBM Consulting. You'll be empowered to challenge the status quo, explore novel ideas, and craft creative solutions that deliver tangible results. Our supportive environment prioritizes your long-term career development while celebrating your unique talents and contributions.
Develop and maintain batch and real-time data pipelines utilizing AWS services like EMR, Glue, Kinesis, and Managed Streaming for Apache Kafka. Create and manage data layers on RedShift, Aurora, and DynamoDB, including data migration using AWS DMS. Schedule and manage data services on the AWS platform for optimal efficiency.
Utilize open-source technologies such as Apache Airflow and dbt, along with Spark (Python/Scala) on AWS. Leverage AWS Glue Databrew for data preparation and quality tasks. Implement Lambda functions with Python for data processing and transformation, and employ RedShift Spectrum for data warehousing and analytics.
Experience in designing, building, and operating batch and real-time data pipelines on AWS is essential. Proficiency with AWS services including EMR, Glue, Kinesis, Redshift, Aurora, DynamoDB, DMS, and Lambda is required. Familiarity with open-source technologies like Apache Airflow, dbt, and Spark (Python/Scala) is crucial.
Demonstrated experience with AWS Glue Databrew for data preparation and quality, Redshift Spectrum for warehousing and analytics, and Lambda functions for data processing is expected. Experience with Managed Streaming for Apache Kafka is also beneficial for developing robust data solutions.
IBM
IT Consulting