Databricks Architect

Persistent

12–15 yrs Pune Full Time Hybrid (office + remote)
Persistent logo
Posted : today
Actively hiring

Job description

Persistent is a leading Digital Engineering and Enterprise Modernization partner, leveraging AI and platform-driven solutions to empower clients and drive innovation. We collaborate with global industry leaders, including Fortune 50 companies and top banks in the US and India. Our commitment to client success and agility has fueled our consistent growth, with over 26,000 global team members and a recent recognition as the fastest-growing Indian IT services brand.

This role seeks an experienced Databricks Architect to design, build, and maintain robust, scalable data pipelines on the Databricks platform, playing a pivotal role in our clients' data strategies.

Responsibilities

Develop and manage comprehensive ETL/ELT pipelines using Databricks, incorporating PySpark, SQL, and Notebooks. Design end-to-end Lakehouse architectures, covering data ingestion, transformation, modeling, and consumption layers.

Implement the Medallion Architecture (Bronze/Silver/Gold) leveraging Delta Lake. Construct high-performance ETL/ELT pipelines utilizing PySpark, Spark SQL, and Delta Lake. Architect and optimize both batch and streaming data pipelines.

Focus on advanced Spark optimization, efficient cluster sizing, and effective cost management. Integrate CI/CD, version control, and DevOps best practices within the Databricks environment.

Qualifications

Possess advanced expertise in the Databricks Lakehouse Platform, PySpark, Spark SQL, Delta Lake, Structured Streaming, and Unity Catalog with its governance frameworks.

Demonstrate strong experience with a major cloud platform such as Azure, AWS, or GCP. Possess a solid understanding of distributed computing, Spark internals, data warehousing concepts, and various data modeling techniques (dimensional, normalized, Lakehouse).

Showcase experience in ETL/ELT architecture, CI/CD pipelines, Git, and orchestration tools. Proven experience in leading large-scale data platform implementations is essential for this role.

Essential Skills

Databricks Lakehouse PlatformPySparkSpark SQLDelta LakeStructured StreamingUnity CatalogAzureAWSGCPDistributed ComputingSpark InternalsData WarehousingData ModelingETL/ELT ArchitectureCI/CDGitOrchestration Tools

Highlights

  • Actively hiring

More Details

RoleDatabricks Architect
IndustryInformation Technology & Services
DepartmentData Architect
Employment TypeFull Time, Hybrid (office + remote)

About the Company

Persistent Systems logo

Persistent Systems

Information Technology & Services

Databricks Architect at Persistent | SkillMX | SkillMX