Senior Azure Data Engineer
Persistent
Persistent
Persistent, a leading AI-driven digital engineering partner, is seeking a seasoned Senior Data Engineer. This role focuses on building, maintaining, and optimizing large-scale data platforms, leveraging extensive experience with Databricks and PySpark. You will play a crucial role in ensuring data integrity and platform stability within enterprise environments.
We are committed to client success and innovation, working with numerous industry leaders globally. Our commitment to growth and talent development is reflected in our recent achievements and recognition. Join our dynamic team and contribute to transforming industries with cutting-edge data solutions.
Your responsibilities will include designing and developing scalable data pipelines using Databricks and PySpark. You will investigate and resolve complex data quality issues, pipeline failures, and production incidents, performing root cause analysis. Monitoring and optimizing Spark workloads for performance and reliability are key. You will also collaborate with cross-functional teams, support production deployments, and establish best practices for data engineering excellence.
Driving continuous improvements in data processing frameworks and platform stability will be a core part of your role. You will also be expected to mentor junior team members and share your expertise effectively. This role requires a proactive approach to problem-solving and a commitment to operational excellence.
This position requires a minimum of 8 years of experience in Data Engineering and Big Data technologies, with strong hands-on expertise in Databricks and PySpark. Proven experience in analyzing and resolving production support issues, framework failures, and data discrepancies is essential. A solid understanding of SQL and data analysis, coupled with experience in complex ETL/ELT pipelines and distributed data processing, is necessary.
Proficiency in troubleshooting, debugging, and performance tuning is critical. Experience working with cloud platforms such as Azure or AWS is a must. We value strong analytical skills, the ability to manage multiple priorities in a fast-paced setting, and excellent communication and collaboration skills. Preferred skills include Delta Lake, Spark Optimization, Data Governance, CI/CD, DevOps, and Data Warehousing/Lakehouse architecture.
Persistent Systems
Information Technology & Services