Databricks Developer
Persistent
Persistent
Join a leading Digital Engineering and Enterprise Modernization partner renowned for its AI-led, platform-driven approach. We empower clients to anticipate future trends and gain a competitive edge through deep technical expertise and industry insights.
We collaborate with industry titans, including a significant portion of Fortune 50 companies, top US banks, and major technology firms. Our culture fosters innovation and agility, driving consistent growth and market leadership.
This role is for a seasoned Databricks Developer/Lead with extensive experience in Data Engineering, Analytics, and Cloud Data Platforms. The ideal candidate will bring profound expertise in Databricks, Apache Spark, distributed data processing, and cloud-native data architectures.
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and Notebooks. Build and manage data processing workflows across Bronze, Silver, and Gold layers utilizing Delta Lake architecture. Develop robust batch and real-time streaming data pipelines with Spark Structured Streaming. Optimize Spark jobs, clusters, and workloads for peak performance, scalability, and cost-efficiency. Implement and maintain comprehensive data quality checks, validation frameworks, and monitoring processes. Collaborate closely with data architects, analysts, and business stakeholders to translate requirements into effective data solutions. Develop and maintain reusable data engineering frameworks and components for enhanced efficiency. Integrate Databricks solutions seamlessly with cloud-native services on Azure, AWS, or GCP. Manage Databricks cluster configurations, autoscaling, job orchestration, and workload optimization. Implement CI/CD pipelines and code deployment processes using Git and DevOps best practices. Troubleshoot and resolve complex issues related to data pipelines, cluster performance, storage, and platform environments. Support production deployments, ensuring platform reliability and operational stability. Contribute actively to architecture discussions and propose enhancements for data platform design and implementation. Mentor junior team members, sharing best practices in Databricks and Spark development. Drive continuous improvement initiatives across the entire data engineering ecosystem.
Possess 7 to 12 years of experience in Data Engineering, Big Data, Analytics, or Data Platform development. Demonstrate hands-on experience delivering multiple enterprise-scale Databricks implementations and migration projects. Exhibit strong expertise in Databricks workspace administration, job orchestration, and notebook development. Showcase extensive experience with Apache Spark, PySpark, Spark SQL, and distributed computing concepts. Possess a deep understanding of Spark internals, execution plans, partitioning, caching, and optimization techniques. Have strong experience implementing Delta Lake architecture and Medallion data design patterns. Show hands-on experience building both batch and streaming data processing solutions. Demonstrate experience with cloud platforms like Azure, AWS, or GCP, with strong expertise in at least one. Exhibit strong knowledge of Databricks cluster management, autoscaling, and resource optimization. Have experience designing and implementing large-scale data lakes and modern data architectures. Be proficient in SQL development, query optimization, and performance tuning. Show experience with Git, CI/CD pipelines, DevOps practices, and automated deployments. Possess a strong understanding of data governance, security, access controls, and compliance standards. Have experience working with structured, semi-structured, and unstructured datasets. Display excellent analytical, debugging, troubleshooting, and problem-solving skills. Communicate effectively and manage stakeholders proficiently. Have experience leading teams, mentoring engineers, and driving technical excellence.
Persistent Systems
Information Technology & Services