Data Engineer-Data Platforms-AWS
IBM
IBM
Embark on a career with IBM Consulting, where innovation thrives on long-term client relationships and global collaboration. You'll partner with leading companies across diverse industries, guiding their hybrid cloud and AI transformations. Leveraging robust IBM technology, Red Hat, and strategic partner support, you'll possess the resources to drive significant change and accelerate client success. Curiosity is the engine of achievement at IBM Consulting; we encourage challenging the status quo, exploring novel ideas, and developing groundbreaking solutions.
Our growth-focused and empathetic culture prioritizes your long-term career development while celebrating your unique talents and contributions. As a Data Engineer specializing in Data Platforms on AWS, you will be instrumental in advising on, developing, and maintaining sophisticated data engineering solutions within the AWS Cloud ecosystem. Your role will involve designing, building, and operating both batch and real-time data pipelines using a variety of AWS services.
Key responsibilities include designing and developing robust data pipelines using AWS services like EMR, Glue, Glue Catalog, and Kinesis, ensuring seamless integration and operational efficiency. You will also create essential data layers on AWS RedShift, Aurora, and DynamoDB, migrating data effectively with AWS DMS.
Furthermore, you will manage data services on the AWS Platform, ensuring optimized performance. The role also involves developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. Proficiency in open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS Platform, is crucial for supporting these data engineering solutions.
Essential experience includes working with AWS services (EMR, Glue, Glue Catalog, Kinesis) for data pipeline development. You should be adept at creating data layers on AWS RedShift, Aurora, and DynamoDB, and migrating data using AWS DMS.
Familiarity with open-source technologies like Apache Airflow and dbt, and Spark/Python or Spark/Scala on AWS is required. Experience in scheduling and managing data services on AWS is also necessary. Exposure to AWS Glue Databrew for data preparation and quality, Lambda functions with Python for data processing and transformation, and RedShift Spectrum for warehousing and analytics are key aspects of this role.
IBM
Information Technology & Services