Senior Azure Data Engineer
Persistent
Persistent
Seeking a skilled Azure Data Engineer with extensive experience in Azure Data Factory and Azure SQL, complemented by practical knowledge of Azure Functions. This role is crucial for architecting, building, and maintaining robust data integration solutions within the Microsoft Azure ecosystem. A deep understanding of data pipelines, ETL/ELT methodologies, SQL development, data transformation, cloud integration, performance tuning, system monitoring, and issue resolution is essential.
Key responsibilities encompass the full lifecycle of data pipelines, from design and development to deployment and ongoing maintenance using Azure Data Factory. You will be instrumental in creating pipelines that efficiently extract and unify data from diverse structured and semi-structured sources.
This position involves developing and managing core ADF components, including pipelines, datasets, linked services, triggers, parameters, and integration runtimes. You will implement both incremental and full data-loading strategies tailored to specific business needs, while also developing reusable, parameterized pipelines to enhance scalability and maintainability. Handling complex data transformations, pipeline orchestration, and robust error-handling scenarios will be a daily focus.
Monitoring ADF pipeline executions and proactively troubleshooting performance bottlenecks or failures is critical. You will implement comprehensive logging, retry mechanisms, exception handling, and operational monitoring to ensure system reliability. The role also requires working with a variety of cloud and on-premises data sources, and supporting the deployment of ADF components across development, testing, and production environments.
In Azure SQL, you will design and develop database solutions, writing and optimizing complex SQL queries, stored procedures, views, functions, and scripts. Query optimization and database performance troubleshooting are key aspects. You will design appropriate database objects and data structures, and develop efficient data ingestion and transformation processes for large datasets. Data validation, reconciliation, and quality assurance will be part of your responsibilities, along with troubleshooting data-related issues across the entire data flow.
Familiarity with indexing, query execution plans, transactions, database security, and performance optimization techniques is expected. You will also support database deployments and production issue resolution.
Azure Functions development will involve creating and maintaining serverless, event-driven processing solutions. These functions will support data integration, automation, validation, and custom processing tasks, integrating seamlessly with ADF and other Azure services. Development will be in languages like C#, .NET, or Python, with a focus on robust exception handling, logging, configuration, and monitoring. You will also develop and consume REST APIs for system-to-system integration.
The ideal candidate possesses strong hands-on experience in Azure Data Factory and Azure SQL, with good experience in Azure Functions. A solid grasp of SQL, relational databases, and cloud-based data engineering concepts is fundamental. You should be adept at designing ETL/ELT pipelines, data integration, transformation, and orchestration, with strong debugging and troubleshooting skills. Experience implementing error handling, logging, and monitoring is required.
Beneficial skills include experience with Azure Blob Storage, Azure Data Lake Storage, Azure Key Vault, Azure Logic Apps, Azure Storage Accounts, Azure DevOps, CI/CD, REST APIs, JSON, PowerShell, Python, Microsoft Fabric, Azure Synapse Analytics, Git, Managed Identity, Azure security concepts, and monitoring tools like Azure Monitor and Application Insights.
We also expect experience with Git-based source control and an understanding of CI/CD for Azure solutions. Proficiency in promoting and deploying ADF and related Azure components across environments is important. Adherence to coding, naming, documentation, and development best practices is a must. Participation in code/design reviews to ensure production-ready solutions is also expected.
- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory. - Build pipelines for extracting and integrating data from various sources. - Develop and manage ADF components: pipelines, datasets, linked services, triggers, parameters, and integration runtimes. - Implement incremental and full data-loading strategies. - Develop reusable and parameterized pipelines for enhanced maintainability. - Handle data transformations, dependencies, orchestration, and error scenarios. - Monitor ADF pipeline execution and troubleshoot failures or performance issues. - Implement logging, retry mechanisms, exception handling, and operational monitoring. - Work with cloud and on-premises data sources. - Support deployment of ADF components across environments. - Design and develop database solutions using Azure SQL Database. - Write and optimize complex SQL queries, stored procedures, views, functions, and scripts. - Troubleshoot database performance issues. - Design database objects and data structures. - Develop efficient data ingestion and transformation processes for large datasets. - Implement data validation, reconciliation, and quality checks. - Troubleshoot data-related issues across source, transformation, and target layers. - Understand indexing, query execution plans, transactions, security, and performance optimization. - Support database deployments and production issues. - Develop and maintain Azure Functions for event-driven processing. - Integrate Azure Functions with ADF and other Azure services. - Develop functions using C#, .NET, Python, or other applicable languages. - Implement exception handling, logging, configuration, and monitoring for Azure Functions. - Develop and consume REST APIs for system-to-system integration. - Work with Git-based source control. - Understand CI/CD concepts for Azure solutions. - Promote/deploy ADF and related Azure components across environments. - Follow coding, naming, documentation, and development best practices. - Participate in code/design reviews.
- Strong hands-on expertise in Azure Data Factory and Azure SQL. - Working experience with Azure Functions. - Solid understanding of data pipelines, ETL/ELT processes, SQL development, data transformation, and cloud integration. - Proficiency in performance optimization, monitoring, and troubleshooting. - Experience designing, developing, and maintaining scalable data integration solutions. - Ability to extract and integrate data from multiple structured and semi-structured data sources. - Experience developing reusable and parameterized pipelines. - Strong capabilities in handling data transformations, dependencies, pipeline orchestration, and error-handling scenarios. - Experience in monitoring ADF pipeline execution and troubleshooting failures. - Ability to implement logging, retry mechanisms, exception handling, and operational monitoring. - Experience working with cloud and on-premises data sources. - Familiarity with Azure SQL Database design and development. - Skill in writing and optimizing complex SQL queries, stored procedures, views, and functions. - Experience in query optimization and troubleshooting database performance issues. - Ability to design appropriate database objects and data structures. - Experience with large datasets and developing efficient data ingestion and transformation processes. - Skill in implementing data validation, reconciliation, and quality checks. - Ability to troubleshoot data-related issues across source, transformation, and target layers. - Understanding of indexing, query execution plans, transactions, database security, and performance optimization. - Experience developing and maintaining Azure Functions for event-driven and serverless processing. - Experience integrating Azure Functions with ADF and other Azure services. - Proficiency in C#, .NET, Python, or other applicable supported languages for Azure Functions development. - Experience implementing appropriate exception handling, logging, configuration, and monitoring for Azure Functions. - Ability to develop and consume REST APIs where required. - Experience working with Git-based source control. - Understanding of CI/CD concepts for Azure solutions. - Experience promoting/deploying ADF and related Azure components across environments. - Adherence to coding, naming, documentation, and development best practices. - Willingness to participate in code/design reviews.
Persistent Systems
IT Consulting