Senior Azure Data Engineer

Persistent

8+ yrs SANTA CLARA Full Time Hybrid (office + remote)
Persistent logo
Posted : today
Actively hiring

Job description

Persistent, a leading AI-driven digital engineering partner, is seeking a seasoned Senior Data Engineer. This role focuses on building, maintaining, and optimizing large-scale data platforms, leveraging extensive experience with Databricks and PySpark. You will play a crucial role in ensuring data integrity and platform stability within enterprise environments.

We are committed to client success and innovation, working with numerous industry leaders globally. Our commitment to growth and talent development is reflected in our recent achievements and recognition. Join our dynamic team and contribute to transforming industries with cutting-edge data solutions.

Responsibilities

Your responsibilities will include designing and developing scalable data pipelines using Databricks and PySpark. You will investigate and resolve complex data quality issues, pipeline failures, and production incidents, performing root cause analysis. Monitoring and optimizing Spark workloads for performance and reliability are key. You will also collaborate with cross-functional teams, support production deployments, and establish best practices for data engineering excellence.

Driving continuous improvements in data processing frameworks and platform stability will be a core part of your role. You will also be expected to mentor junior team members and share your expertise effectively. This role requires a proactive approach to problem-solving and a commitment to operational excellence.

Qualifications

This position requires a minimum of 8 years of experience in Data Engineering and Big Data technologies, with strong hands-on expertise in Databricks and PySpark. Proven experience in analyzing and resolving production support issues, framework failures, and data discrepancies is essential. A solid understanding of SQL and data analysis, coupled with experience in complex ETL/ELT pipelines and distributed data processing, is necessary.

Proficiency in troubleshooting, debugging, and performance tuning is critical. Experience working with cloud platforms such as Azure or AWS is a must. We value strong analytical skills, the ability to manage multiple priorities in a fast-paced setting, and excellent communication and collaboration skills. Preferred skills include Delta Lake, Spark Optimization, Data Governance, CI/CD, DevOps, and Data Warehousing/Lakehouse architecture.

Essential Skills

AzureDatabricksPySparkSQLETLELTCloud Platforms

Good to Have

Delta LakeSpark OptimizationData GovernanceCI/CDDevOpsData WarehousingLakehouse Architecture

Highlights

  • Actively hiring

More Details

RoleSenior Azure Data Engineer
IndustryInformation Technology & Services, Data Analytics, Software Development
DepartmentData Engineering, Software Development
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Persistent Systems logo

Persistent Systems

Information Technology & Services

Senior Azure Data Engineer at Persistent | SkillMX | SkillMX