Data Engineer-Data Platforms-Azure
IBM
IBM
Join IBM Consulting and shape the future of hybrid cloud and AI for leading companies worldwide. You'll collaborate closely with clients, leveraging IBM technology and Red Hat to drive impactful change. Our culture fosters curiosity, innovation, and continuous growth, supporting your long-term career development while valuing your unique contributions.
As a Data Engineer on the Data Platforms team, you will be instrumental in developing and managing sophisticated data engineering solutions within the Azure cloud environment.
Your core responsibilities will involve designing, constructing, and managing robust batch and real-time data pipelines utilizing a suite of Azure services. This includes building and operating the data layer on Azure Synapse Analytics, SQL DW, and Cosmos DB. You'll also implement and maintain key Azure Data Platform components such as ADLS2, Blob Storage, Synapse Analytics with Spark and SQL, Azure Functions with Python, Azure Purview, and Cosmos DB.
Furthermore, you will apply your expertise in open-source technologies like Apache Airflow and dbt, alongside Spark with Python or Scala, to enhance data engineering solutions. Ensuring the seamless operation and maintenance of these data engineering solutions for both batch and real-time processing is crucial.
We are seeking individuals with proven experience in data engineering on the Azure Cloud ecosystem, including the design, development, and operation of batch and real-time data pipelines. A strong understanding of Azure Data Platform components is essential, encompassing ADLS2, Blob Storage, SQLDW, Synapse Analytics (Spark and SQL), Azure Functions (Python), Azure Purview, and Cosmos DB.
Proficiency in open-source technologies such as Apache Airflow and dbt, coupled with Spark (Python or Scala), is required. Experience in designing, building, and operating the data layer on Azure Synapse Analytics, SQL DW, and Cosmos DB is also a key requirement. Familiarity with Azure services like Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, and Event Hub for pipeline development is expected.
IBM
Information Technology & Services