Databricks Architect
Persistent
Persistent
Persistent is a leading Digital Engineering and Enterprise Modernization partner, leveraging AI and platform-driven solutions to empower clients and drive innovation. We collaborate with global industry leaders, including Fortune 50 companies and top banks in the US and India. Our commitment to client success and agility has fueled our consistent growth, with over 26,000 global team members and a recent recognition as the fastest-growing Indian IT services brand.
This role seeks an experienced Databricks Architect to design, build, and maintain robust, scalable data pipelines on the Databricks platform, playing a pivotal role in our clients' data strategies.
Develop and manage comprehensive ETL/ELT pipelines using Databricks, incorporating PySpark, SQL, and Notebooks. Design end-to-end Lakehouse architectures, covering data ingestion, transformation, modeling, and consumption layers.
Implement the Medallion Architecture (Bronze/Silver/Gold) leveraging Delta Lake. Construct high-performance ETL/ELT pipelines utilizing PySpark, Spark SQL, and Delta Lake. Architect and optimize both batch and streaming data pipelines.
Focus on advanced Spark optimization, efficient cluster sizing, and effective cost management. Integrate CI/CD, version control, and DevOps best practices within the Databricks environment.
Possess advanced expertise in the Databricks Lakehouse Platform, PySpark, Spark SQL, Delta Lake, Structured Streaming, and Unity Catalog with its governance frameworks.
Demonstrate strong experience with a major cloud platform such as Azure, AWS, or GCP. Possess a solid understanding of distributed computing, Spark internals, data warehousing concepts, and various data modeling techniques (dimensional, normalized, Lakehouse).
Showcase experience in ETL/ELT architecture, CI/CD pipelines, Git, and orchestration tools. Proven experience in leading large-scale data platform implementations is essential for this role.
Persistent Systems
Information Technology & Services