Lead Databricks Data Engineer
Persistent
Persistent
Join Persistent as a Lead Databricks Data Engineer and drive the future of data solutions. This role focuses on designing, developing, and maintaining scalable data pipelines and cloud-native platforms using the Databricks Lakehouse Platform.
Leverage your expertise in Big Data technologies and enterprise analytics to build robust, high-performance data solutions that empower data-driven decision-making across the organization. You will collaborate with cross-functional teams to deliver innovative and reliable data products.
This is an opportunity to work with cutting-edge technologies in a dynamic and growth-oriented environment, contributing to industry-leading projects.
Design and implement scalable ETL/ELT pipelines using Databricks and PySpark.
Build and optimize large-scale data processing workflows and develop data ingestion frameworks.
Develop real-time and batch data processing solutions, supporting Delta Lake and Lakehouse architectures.
Optimize Databricks workloads for performance, cost, and reliability, while implementing robust data validation and cleansing processes.
Collaborate with architects and data scientists to deliver business-driven data solutions and support production deployments.
Bring 8-12 years of hands-on experience in Data Engineering, with advanced proficiency in Databricks Lakehouse Platform, PySpark, Spark SQL, Python, and SQL.
Possess extensive experience in designing scalable ETL/ELT pipelines, data integration, and Delta Lake architecture.
Demonstrate strong knowledge of Data Warehousing concepts, dimensional modeling, and experience with cloud platforms (Azure, AWS, GCP).
Familiarity with Git, CI/CD, and experience in mentoring junior engineers is essential. Strong analytical, communication, and stakeholder management skills are required.
Persistent Systems
Information Technology & Services