Senior Azure Data Engineer
Persistent
Persistent
Join a leading AI-driven digital engineering partner focused on modernizing enterprises and driving digital transformation. We collaborate with global industry leaders, including Fortune 50 companies and top financial institutions in the US and India. Our commitment to innovation, client success, and agility fuels our sustained growth. We are recognized for our leadership in IT services and private equity services, consistently achieving strong revenue growth and expanding our global team.
This role is for an experienced Senior Azure Data Engineer, a pivotal position in designing and developing robust cloud-based data platforms. You will be instrumental in building advanced data architectures, creating enterprise-grade ETL/ELT pipelines, and empowering data-driven decision-making throughout the organization. The position demands hands-on expertise with Azure cloud services, Snowflake data warehousing, and large-scale data processing frameworks.
Design, develop, and maintain scalable data pipelines utilizing Azure Databricks and PySpark. Build and optimize ETL/ELT workflows for diverse datasets, including structured, semi-structured, and unstructured data. Develop and manage Snowflake data warehouse solutions, focusing on efficient data ingestion, transformation, and performance tuning. Create reusable frameworks for data ingestion, orchestration, transformation, validation, and monitoring processes. Integrate data from various enterprise applications, databases, APIs, and cloud platforms into Azure Data Lake and Snowflake. Design and implement modern data lake and Lakehouse architectures. Optimize data processing workloads for enhanced performance, reliability, scalability, and cost-effectiveness. Implement best practices for data quality, governance, security, and monitoring. Collaborate closely with business stakeholders, solution architects, data analysts, and engineering teams to deliver impactful data solutions. Troubleshoot and resolve production issues impacting data pipelines and cloud data platforms. Participate in code reviews, CI/CD implementation, deployment automation, and DevOps initiatives. Support data platform modernization efforts and drive continuous improvement. Contribute to architecture discussions and propose best practices for scalable cloud data solutions.
Minimum 6 to 10 years of experience in Data Engineering and Cloud Data Platform development.
Expertise includes: - Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake. - Proven proficiency in Snowflake, covering data loading, transformation, query optimization, data sharing, and performance tuning. - Advanced SQL skills for complex query development, optimization, troubleshooting, and performance management. - Experience with Azure services like Data Factory (ADF), Data Lake Storage (ADLS Gen2), and Key Vault, along with Azure DevOps. - Solid understanding of Data Warehousing principles, Dimensional Modeling, ETL/ELT frameworks, and Lakehouse architecture. - Experience building and supporting enterprise-scale data pipelines and analytics platforms. - Knowledge of data governance, data quality, metadata management, and security best practices. - Familiarity with Git, CI/CD pipelines, source control management, and Agile methodologies. - Strong grasp of distributed computing and cloud-native data architectures. - Excellent analytical, troubleshooting, debugging, and problem-solving capabilities. - Superior communication and stakeholder management skills. - Ability to work autonomously while effectively collaborating with distributed teams.
Persistent Systems
IT Consulting