Lead Databricks Data Engineer

Persistent

8–12 yrs Bengaluru, Pune Full Time Hybrid (office + remote)
Persistent logo
Posted : today
Actively hiring

Job description

Join Persistent as a Lead Databricks Data Engineer and drive the future of data solutions. This role focuses on designing, developing, and maintaining scalable data pipelines and cloud-native platforms using the Databricks Lakehouse Platform.

Leverage your expertise in Big Data technologies and enterprise analytics to build robust, high-performance data solutions that empower data-driven decision-making across the organization. You will collaborate with cross-functional teams to deliver innovative and reliable data products.

This is an opportunity to work with cutting-edge technologies in a dynamic and growth-oriented environment, contributing to industry-leading projects.

Responsibilities

Design and implement scalable ETL/ELT pipelines using Databricks and PySpark.

Build and optimize large-scale data processing workflows and develop data ingestion frameworks.

Develop real-time and batch data processing solutions, supporting Delta Lake and Lakehouse architectures.

Optimize Databricks workloads for performance, cost, and reliability, while implementing robust data validation and cleansing processes.

Collaborate with architects and data scientists to deliver business-driven data solutions and support production deployments.

Qualifications

Bring 8-12 years of hands-on experience in Data Engineering, with advanced proficiency in Databricks Lakehouse Platform, PySpark, Spark SQL, Python, and SQL.

Possess extensive experience in designing scalable ETL/ELT pipelines, data integration, and Delta Lake architecture.

Demonstrate strong knowledge of Data Warehousing concepts, dimensional modeling, and experience with cloud platforms (Azure, AWS, GCP).

Familiarity with Git, CI/CD, and experience in mentoring junior engineers is essential. Strong analytical, communication, and stakeholder management skills are required.

Essential Skills

DatabricksData EngineeringBig DataPySparkSpark SQLPythonSQLETLELTDelta LakeData WarehousingDimensional ModelingCloud PlatformsAzureAWSGCPGitCI/CDKafkaUnity CatalogData GovernanceSchema DesignData ModelingQuery OptimizationAzure DevOpsAI/ML Data PipelinesFeature Engineering

Good to Have

Azure Data FactoryAWS GlueEvent HubDeployment Automation

Highlights

  • Actively hiring

More Details

RoleLead Databricks Data Engineer
IndustryInformation Technology & Services
DepartmentData Engineering, Data & Analytics
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Persistent Systems logo

Persistent Systems

Information Technology & Services

Lead Databricks Data Engineer at Persistent | SkillMX | SkillMX