Data Engineer-Data Platforms-AWS

IBM

Fresher Bengaluru Full Time Hybrid (office + remote)
IBM logo
Posted : today
Actively hiring

Job description

Join IBM Consulting and shape the future of hybrid cloud and AI for leading companies worldwide. As a Data Engineer specializing in Data Platforms on AWS, you will be instrumental in designing, developing, and maintaining robust data engineering solutions within the AWS Cloud ecosystem. This role offers a unique opportunity to collaborate with clients and leverage cutting-edge IBM technology and Red Hat solutions.

At IBM Consulting, curiosity and innovation are at the core of our success. We encourage challenging the status quo, exploring novel ideas, and creating impactful solutions that drive tangible results. Our supportive culture fosters long-term career development, valuing your distinct skills and experiences as you contribute to client success.

Responsibilities

Your primary responsibilities will include designing and building scalable batch and real-time data pipelines utilizing key AWS services such as EMR, Glue, Glue Catalog, and Kinesis. You will also create and manage data layers on AWS RedShift, Aurora, and DynamoDB, and perform data migrations using AWS DMS.

Develop sophisticated batch and real-time data pipelines for Data Warehouse and Datalake environments, leveraging AWS Kinesis and Managed Streaming for Apache Kafka. Effectively schedule and manage data services on the AWS Platform to ensure seamless operation and integration of all data engineering solutions.

Apply open-source technologies like Apache Airflow and dbt, alongside Spark with Python or Scala on the AWS Platform, to enhance and support your data engineering solutions. This role requires hands-on experience with the complete AWS toolset for data pipeline development and management.

Qualifications

We are seeking individuals with practical experience in developing batch and real-time data pipelines for Data Warehouse and Datalake scenarios, utilizing AWS Kinesis and Managed Streaming for Apache Kafka. Experience in creating data layers using AWS RedShift, Aurora, and DynamoDB, and migrating data with AWS DMS is essential.

Proficiency in applying open-source technologies such as Apache Airflow and dbt, along with Spark/Python or Spark/Scala on the AWS Platform, is required. You should have demonstrated experience working with AWS services including EMR, Glue, Glue Catalog, and Kinesis for pipeline design and operation.

Familiarity with AWS Glue Databrew for data preparation and quality, knowledge of Lambda functions with Python for data processing, and experience with RedShift Spectrum for data warehousing and analytics are highly desirable. You will be responsible for scheduling and managing data services on the AWS Platform.

Essential Skills

AWSPythonSparkScalaData EngineeringData PipelinesData WarehousingData IntegrationSchedulingAmazon RedshiftAmazon DynamoDBApache KafkaAWS GlueAWS KinesisAWS EMRApache Airflow

Good to Have

dbtAWS DMSAWS LambdaRedshift SpectrumAWS Glue DatabrewManaged Streaming for Apache Kafka

Highlights

  • Actively hiring

More Details

RoleData Engineer-Data Platforms-AWS
IndustryInformation Technology & Services
DepartmentEngineering, Technology
Employment TypeFull Time, Hybrid (office + remote)

About the Company

IBM logo

IBM

Information Technology & Services

Data Engineer-Data Platforms-AWS at IBM | SkillMX | SkillMX